Build a REST API in Go (Golang)

75% of graduated [CNCF projects](https://www.cncf.io/projects/) use Go (Golang). Notable cloud projects that use Go include Kubernetes, Docker, Helm, Prometheus and consul. Some of these projects expose Go to the surface. `kubectl` allows you to use Go templates. Helm uses Go templating also in its yaml files. As a result, I've been looking into Go. So let's build a server in Go (Golang). This is like the "hello world" of backend development.

## Getting started

First, create a module.

```bash
go mod init mudassir.dev
```

Now, let's return `Hello World!` from our endpoint.

```go
// File: main.go
package main

import (
  "net/http",
  "log",
  "fmt"
)

func main() {
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintf(w, "Hello World!")
	})

  log.Fatal(http.ListenAndServe(":8080", nil))

}
```

Start the server using `go run main.go`. Running `curl -X GET localhost:8080` will return `Hello World!`. That was quite simple to do.

Couple of things are going on here. We're importing the `net/http` package. We're then registering an endpoint with a handler function. This handler function is an anonymous function. We then start the server on port `8080`.

However, we're returning a simple string. Could we have a JSON response instead?

## Adding JSON

Let's update our import to include the `ecoding/json` package.

```go
import (
  "net/http",
  "log",
  "fmt",
  "encoding/json"
)
```

Let's create a struct to hold data.

```go
var greeting struct {
  Message string `json:"message"`
}

greeting.Message = "Hello World!"
```

All together, our code should look like this.

```go
package main

import (
  "fmt"
  "log"
  "net/http"
  "encoding/json"
)

func main() {

  http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
    var greeting struct {
      Message string `json:"message"`
    }

    greeting.Message = "Hello World!"

    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(greeting)
  })

  log.Fatal(http.ListenAndServe(":8080", nil))
}
```

Start the server using `go run main.go`. Now run `curl -X GET localhost:8080`. We will get the following JSON response.

```json
{
  "message": "Hello World!"
}
```

Now let's refactor a little. We can extract the anonymous function. This also improves readability.

```go
// File: main.go
package main

import (
  "fmt"
  "log"
  "net/http"
  "encoding/json"
)

func greetingHandler(w http.ResponseWriter, r *http.Request) {

  var greeting struct {
    Message: string `json:"message"`
  }

  greeting.Message = "Hello World!"

  w.Header().Set("Content-Type", "application/json")
  json.NewEncoder(w).Encode(test)
}

func main() {
  http.HandleFunc("/", greetingHandler)

  log.Fatal(http.ListenAndServe(":8080", nil))
}


```

## What next?

Here's some things you can do next for fun.

- Use the `flag` package to listen on a different port
- Use an environment variable to listen on a different port
- Unit tests
- Add a database
- Use the repository pattern
- Document your API using OpenAPI
- Use gRPC instead of REST
- Event driven architecture using Apache Kafka
- Containerise using Docker
- Add some logs (now read them using [fluentd](https://www.fluentd.org/))

## Useful resources

- [Idiomatic Go](https://go.dev/doc/effective_go)
- [Go net/http package](https://pkg.go.dev/net/http)
- [Go by Example](https://gobyexample.com/)
- [The Go standard library](https://pkg.go.dev/std)
- [The Zen of Go](https://dave.cheney.net/2020/02/23/the-zen-of-go)
- [Go at Google](https://go.dev/talks/2012/splash.article)


Kubernetes (k8s) cheatsheet

I recently started studying for the CKAD certification, here's my cheat sheet.

## Using minikube

[Minikube](https://minikube.sigs.k8s.io/docs/) allows you to run a local instance
of k8s. This is great for experimenting without the costs involved with cloud. 
Alternatively, you can use cloud providers like AWS, GCP or Azure. In the past,
I have also used Digital Ocean's 
[Managed Kubernetes](https://www.digitalocean.com/products/kubernetes).

```bash
# Start minikube
minikube start

# SSH into cluster
minikube ssh

# Connect to LoadBalancer services
minikube tunnel
```

## Use kubectl faster

I suggest you also install [bash completion](https://kubernetes.io/docs/reference/kubectl/cheatsheet/#bash) and the `k` alias.

## Spot the patterns

k get <resource>

## What you should do when you enter the exam?

Put these in your bashrc

```bash
# Find the number of questions (n) then
mkdir {1..n}
# Format question_number%weight status
echo -e {1..n}"%\\n"

alias k=kubectl # will already be pre-configured with autocompletion
alias bashconfig="vim ~/.bashrc" # most important since you can edit your .bashrc quick
alias ..="cd .."
set -o vi # Vim keybindings in bash (or use fc command)

# Short for "dry output"
# k create deploy nginx --image=nginx $do
export do="--dry-run=client -o yaml"   
# Note: I faced this issue in zsh: https://unix.stackexchange.com/questions/19530/expanding-variables-in-zsh

export now="--force --grace-period 0"   # k delete pod x $now

# Switch namespaces fast
alias kn='kubectl config set-context --current --namespace '

kn default # set to default namespace
kn jupiter # set to jupiter namespace

alias what_ns="k config view -o jsonpath='{.contexts[0].context.namespace}'"
```

The following settings will already be configured in your real exam environment
in `~/.vimrc`. But it can never hurt to be able to type these down:

```sh
set tabstop=2
set expandtab
set shiftwidth=2
```

Press `.` to repeat `<` or `>` indent action.

## Use tmux

You can use tmux to split your terminal.

## Containerisation

Docker. Podman. Containderd. CRI-O.

## Staple commands

```bash
# See resources
k api-resources
k api-resourced --namespaced

# Show pods
k get po
k get pod

# Show services
k get svc
k get service

# Show together
k get pod,svc

# Generate job
k create job sleepy --image=busybox --dry-run=client -o yaml

# Create CronJob
# https://crontab.guru/every-5-minutes
# "*/5 * * * *"

# Update image
kubectl set image deployments/kubernetes-bootcamp kubernetes-bootcamp=jocatalin/kubernetes-bootcamp:v2

# kubectl set image POD/POD_NAME CONTAINER_NAME=IMAGE_NAME:TAG
kubectl set image pod/nginx nginx=nginx:1.7.1

# History
k rollout history deployments deployment-name

# Events
k get events

# create pod
k run my-pod --image=nginx

# Help
k exec --help
k exec -h

# Create a secret
k create secret generic test --from-literal=entree=meatloaf --dry-run=client -o yaml

# aliases
alias k=kubectl
alias kd='kubectl describe'

# Usage
k create secret generic lfsecret --from-literal=password=LFTr@1n $do

# Use yq to get node
k create secret generic lfsecret --from-literal=password=LFTr@1n $do | yq .data.password

# Explain
k explain cronjob.spec --recursive
k explain cronjib.spec --recursive | grep -i failed -C20

# Find other ways to do the same command
kubectl set resources deployment/nginx-deployment -c=nginx --limits=cpu=200m,memory=512Mi

# Print out all the environment variables in a pod - useful when using secrets
k exec -i pod-name -- env

# Run a temporary pod to curl a pod
k run tmp --image=nginx:alpine -i --rm --restart=Never -- curl 172.17.0.10
k run tmp --restart=Never --rm -i --image=nginx:alpine -- curl 10.44.0.78 # From killer.sh

## Run interactive
kubectl run -i --tty busybox --image=busybox --restart=Never -- sh

## Get Node IP
k get nodes -o wide

k get endpoints

# Write to stdout - useful for busybox containers
wget -O - localhost:8080
```

```bash
k -n neptune get rs -o wide

k config view | grep namespace -C 10 # context 10 lines

k config view -o jsonpath='{.contexts[0].context.namespace}'

man 5 crontab # get cron job expression examples

# See liveness probe syntax and keep color from grep
# If there's one thing you take from this blog post - it should be this
k explain pod.spec --recursive | grep liveness -C 10 --color=always | less -r
k explain pod.meta
k explain pod.spec.containers.volumeMounts
k explain pod.spec.volumes.secret
k explain pod.spec.containers.envFrom.secretRef

```

```bash
docker build -t my-image-name:latest . # build the container
docker images # see image
docker run my-image-name:latest # run the image
docker push # push to the repo

kubectl create deployment <Deploy-Name> --image=<repo>/<app-name>:<version>
kubectl create deployment time-date --image=10.110.186.162:5000/simpleapp:v2.2

# Running commands in a container
kubectl exec -i​t <Pod-Name> -- /bin/bash

export repo=10.97.40.62:5000 # use variables to be faster
curl $repo/v2/_catalog

# Edit to allow HTTP repo in podman, set insecure to true or --tls-verify=false
# https://projectatomic.io/blog/2018/05/podman-tls/ 
sudo vim /etc/containers/registries.conf
sudo systemctl restart crio # optional
sudo systemctl status crio # optional


# Get pod names - useful if u want to loop through and do something
k get pods -o name

# bash for loop
for item in 1 2 3
do
  echo $item
done

# scale a deployment
k scale deployment <deployment-name> --replicas=5

# Other useful bash commands
tail # good for looking at logs
head
less

# list annotations
kubectl annotate pod nginx1 --list


```

## Working with labels

Working with labels is just like work with sets in Mathematics. 

```bash
# working with labels - it's like working with sets

# All pods with a label run - value not checked
k get pods -l 'run'

# All pods without a label run - value not checked
k get pods -l '!run'

# All pods with label environment=test or environment=qa
k get pods -l 'environment in (test, qa)'

# All pods with label environment=test or environment=qa and run=app
k get pods -l 'environment in (test, qa), run in (app)'

# All pods not in environment=test
k get pods -l 'environment notin (test)'

# Get pods with label columns
k get pods -L environment

# tip
seq 2 5 # 2 3 4 5
seq 3 # 1 2 3

# Delete all pods
kubectl delete po --all

# Find data faster
k get pod nginx -o json | vim # now you can navigate around the JSON quick 
# and figure out the JSON path

# Get the YAML for a new ResourceQuota called 'myrq' with hard limits of 1 CPU, 1G memory and 2 pods without creating it
kubectl create quota myrq --hard=cpu=1,memory=1G,pods=2 --dry-run=client -o yaml
```

Remember to copy files over as requested in the question
Remember to delete pods that are no longer needed
Make sure you name the containers properly when asked in a question

You should be familar with https://helm.sh/docs/intro/quickstart/

TIP: Use `<C-R>` to invoke reverse search in bash

Tip: You need to learn how to use vim buffers
Tip: Tmux is good for having mutliple terminals - one for docs and one for vim

tmux

<ctrl-b> then `:set-window-option mode-keys vi`

fullscreen: ctrl-b then z


## FAQ

> **Waning**: you should check the curriculum as it changes often and
> the answers here may be out of date.

### What score is needed to pass the exam?

For the CKAD Exam, a score of 66% or above must be earned to pass.

### What you have?

For your convenience, all environments, in other words, the base system and the
cluster nodes, have the following additional command-line tools pre-installed
and pre-configured:

- `kubectl` with `k` alias and Bash autocompletion
- `yq` and `jq` for YAML/JSON processing
- `tmux` for terminal multiplexing
- `curl` and `wget` for testing web services
- `man` and man pages for further documentation

## References
- https://kubernetes.io/
- https://kubernetes.io/docs/reference/kubectl/cheatsheet/
- [Kubectl reference docs](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands)

## Practice!
- [Practice enough with these 150 questions for the CKAD exam](https://medium.com/bb-tutorials-and-thoughts/practice-enough-with-these-questions-for-the-ckad-exam-2f42d1228552)
- https://github.com/dgkanatsios/CKAD-exercises

## CKAD Official Links
- https://github.com/cncf/curriculum
- [CKAD Curriculum for exam](https://github.com/cncf/curriculum/blob/master/CKAD_Curriculum_v1.25.pdf)
- https://docs.linuxfoundation.org/tc-docs/certification/tips-cka-and-ckad

## CKAD Allowed domains and their subdomains
- https://helm.sh/docs/
- https://kubernetes.io/docs/
- https://kubernetes.io/blog/

## Helpful tips from previous students
- [Tips and tricks](https://kodekloud.com/community/t/thanks-a-lot-mumshad-mannambeth-and-kodekloud-team-got-through-the-exam-and-sc/158295)
- [CKAD Bookmarks](https://github.com/reetasingh/CKAD-Bookmarks)

## Other useful links
- https://github.com/json-path/JsonPath
- https://tmuxcheatsheet.com/


Mozilla's DeepSpeech an open source speech to text engine

In this article, we will be trying a transcriber made using Mozilla's DeepSpeech.

## Installation

Let's start with an example.

```bash
# Create and activate a virtualenv
virtualenv -p python3 $HOME/tmp/deepspeech-venv/
source $HOME/tmp/deepspeech-venv/bin/activate

# Install DeepSpeech
pip3 install deepspeech

# Download pre-trained English model files
curl -LO https://github.com/mozilla/DeepSpeech/releases/download/v0.7.3/deepspeech-0.7.3-models.pbmm
curl -LO https://github.com/mozilla/DeepSpeech/releases/download/v0.7.3/deepspeech-0.7.3-models.scorer
```

## Demo - transcribing an audio file

Now let's transcribe an audio file.

```bash
# Download example audio files
curl -LO https://github.com/mozilla/DeepSpeech/releases/download/v0.7.3/audio-0.7.3.tar.gz
tar xvf audio-0.7.3.tar.gz

# Transcribe an audio file
deepspeech --model deepspeech-0.7.3-models.pbmm --scorer deepspeech-0.7.3-models.scorer --audio audio/2830-3980-0043.wav
```

The output should look similar to the verbose below. Notice the line
"experience proves this" which shows this is working.

```
Loading model from file deepspeech-0.7.3-models.pbmm
TensorFlow: v1.15.0-24-gceb46aa
DeepSpeech: v0.7.3-0-g8858494
Loaded model in 0.00997s.
Loading scorer from files deepspeech-0.7.3-models.scorer
Loaded scorer in 0.000207s.
Running inference.
experience proves this
Inference took 1.007s for 1.975s audio file.
```

The last line shows something note worthy - inference took less
time than the audio file length.

## Streaming to DeepSpeech

Make sure you have the [prerequisites](https://github.com/mozilla/DeepSpeech-examples/tree/r0.7/nodejs_mic_vad_streaming#prerequisites) before continuing.

```bash
# Prerequisites
wget https://github.com/mozilla/DeepSpeech/releases/download/v0.7.0/deepspeech-0.7.0-models.pbmm
wget https://github.com/mozilla/DeepSpeech/releases/download/v0.7.0/deepspeech-0.7.0-models.scorer

# Dependancies
sudo apt-get install libasound2-dev

# Clone and move into working directory
git clone https://github.com/mozilla/DeepSpeech-examples moz-ds-examples
cd moz-ds-examples/nodejs_mic_vad_streaming

# Install npm packages
npm install

# Start the server
node start.js
```

I had an issue where no verboise was given. This is because it uses the default
mic which may not exist.

So update `microphone` variable to the following - notice device property was
added to the [options](https://www.npmjs.com/package/mic#micoptions). Run
`arecord -l` to see the list of microphones installed.

```javascript
var microphone = mic({
  rate: "16000",
  channels: "1",
  debug: false,
  fileType: "wav",
  device: "plughw:2,0",
});
```

Now start streaming.

## Next steps

Mozilla's DeepSpeech has an API in C,.NET, Java, JavaScript (NodeJS/ElectronJS)
and Python so i'm sure you could integrate this the next time you need
speech-to-text - I know I will be. You may want to train your own models. You
could use this for an IoT device.


Getting started with GNU Octave (or MATLAB) for machine learning

82% of Fortune 100 companies use MATLAB; Octave allows you to prototype your
Machine Learning experiments faster, saving you time and money. Since it's a
high level programming language you can do things really fast. Some people
prefer to get their algorithms working in MATLAB and then migrate to another
language such as Python, R, Java or C++. So learning to use MATLAB is a good
investment.

> MathWorks provides a free online self-paced [MATLAB
> Course](https://matlabacademy.mathworks.com/). You also get a certificate which
> you can share to show others you have completed the course. The course is
> really well made.

## Installation

Since MATLAB has a price tag, we will use an open source alternative called
[Octave](https://octave.org). Octave is largely compatible with MATLAB - so you
can still follow along if your're a MATLAB user.

On Ubuntu 20.04

```bash
sudo apt update
sudo apt install octave
```

For other distributions, please consult [GNU Octave
download](https://www.gnu.org/software/octave/#install) website.

## Basics

Instead of replicating a tutorial, below are commands you will find helpful
when running your machine learning experiments.

| Command(s)                            | Usage                                                                                         |
| :------------------------------------ | :-------------------------------------------------------------------------------------------- |
| `+, -, \\, *`                         | Elementary math operations                                                                    |
| `==`                                  | Logical equal to                                                                              |
| `~=`                                  | Logical not equal to                                                                          |
| `&`                                   | Logical AND                                                                                   |
| `|`                                   | Logical OR                                                                                    |
| `xor(a, b)`                           | Logical XOR                                                                                   |
| `eye(n)`                              | n x n identity matrix                                                                         |
| `ones(n, m)`                          | n x m matrix of ones                                                                          |
| `rand(n, m)`                          | n x m matrix of randoms                                                                       |
| `randi(n, m)`                         | n x m matrix of random integers                                                               |
| `size(A)`                             | Dimensions of matrix $A$                                                                      |
| `clear A`                             | Remove variable $A$                                                                           |
| `clc`                                 | Clear command window                                                                          |
| `A'`                                  | Transpose of matrix $A$                                                                       |
| `A(1,2)`                              | Get element $A_{1,2}$                                                                         |
| `A(2,:)`                              | Get the 2nd row of $A$                                                                        |
| `A(end, :)`                           | Get the last row using the end keyword                                                        |
| `pinv(A)`                             | Pseudo inverse of matrix $A$                                                                  |
| `a = 3`                               | Variable assignment                                                                           |
| `a = 3;`                              | Variable assignment - semicolon suppresses output                                             |
| `pi`                                  | Constant PI                                                                                   |
| `A = [1 2; 3 4; 5 6]`                 | A 3x2 matrix with denoted elements                                                            |
| `V = [1; 2; 3]`                       | A 3x1 column vector i.e $V \in \mathbb{R}^3$                                                  |
| `1:4`                                 | Creates a matrix $\begin{bmatrix}1 & 2 & 3 & 4\end{bmatrix}$                                  |
| `0:2:4`                               | Creates a matrix $\begin{bmatrix}0 & 2 & 4\end{bmatrix}$ values from 0 to 4 with spacing of 2 |
| `linspace(first, last, num_elements)` | Useful when creating a range of values (no need to calculate spacing yourself)                |
| `help eye`                            | Shows man page for eye                                                                        |
| `doc eye`                             | Open doc page for eye                                                                         |
| `pwd`                                 | Print working directory                                                                       |
| `cd`                                  | Change directory                                                                              |
| `ls`                                  | List files                                                                                    |
| `load someData.dat`                   | Load data                                                                                     |
| `save filename variable`              | Save variables                                                                                |
| `save workspace.mat`                  | Save variables in workspace to a MAT-file                                                     |
| `load workspace.mat`                  | Load saved workspace variables                                                                |
| `format long`                         | Increase precision (default is short)                                                         |
| `who`                                 | Show variables in the current scope                                                           |
| `whos`                                | Show variables in the current scope with more info                                            |

Watch out: in Octave/MATLAB vectors are not zero-indexed i.e. they start at one

Note: All MATLAB variables are arrays. Understand the difference between
matrix, column vector, row vectror and scalar. Fun fact: MATLAB is an
abbreviation for MATrix LABoratory - makes sense.

Tip: If you only use one index with a matrix, it will traverse down each column
in order. Using one index, try extracting the eighth element of data.

## Manipulating data (matrices/vectors)

| Command       | Usage                                                                                                                                      |
| :------------ | :----------------------------------------------------------------------------------------------------------------------------------------- |
| `A * B`       | Matrix multiplication                                                                                                                      |
| `A .* B`      | Element-wise multiplication                                                                                                                |
| `abs(V)`      | Element-wise absolute                                                                                                                      |
| `exp(V)`      | Element-wise exponentiation                                                                                                                |
| `log(V)`      | Element-wise log                                                                                                                           |
| `max(A)`      | Maximum element in $A$                                                                                                                     |
| `round(A)`    | Round each element to the nearest integer                                                                                                  |
| `a < 2`       | Element-wise comparison                                                                                                                    |
| `find(A < 3)` | Elements that are less than 3                                                                                                              |
| `sum(A)`      | Sum of $A$                                                                                                                                 |
| `magic(n)`    | Generate a n x n magic matrix                                                                                                              |
| `A .* eye(n)` | Assuming $A$ is an n x n matrix then this yields the diagonal and all other elements are zero                                              |
| `flipud(A)`   | Flip matrix up-down                                                                                                                        |
| `A(A > 2)`    | Logical indexing. You can use a logical array as an array index, in which case MATLAB extracts the array elements where the index is true. |

## Plotting data

Plot the $\sin$ function.

```matlab
x = [0:0.01:0.98];
y = sin(2 * pi * 4 * x);

plot(x, y);
```

Plot the $\sin$ and $\cos$ function on the same graph.

```matlab
% Create an array from 0 to 0.98 in steps of 0.01
x = [0:0.01:0.98];

% Sine function
y_sin = sin(2 * pi * 4 * x); % notice the use of x

% Cosine function
y_cos = cos(2 * pi * 4 * x);

% Multple plots using 'hold on'
plot(x, y_sin);
hold on;
plot(x, y_cos, 'r'); % cos function in red

% Tip: use xlim([xmin xmax]) to zoom in on an area of interest

% Add title, axis labels and legend
title('Plot title')
xlabel('x-value')
ylabel('y-value')
legend('sin', 'cos')

% Save/export the plot as png
print -dpng 'plot.png'

% See plot gallery at https://uk.mathworks.com/products/matlab/plot-gallery.html
```

Use `figure` to create multiple figures. Tip: use `help plot` for more
information on how to use the `plot` command. Use `subplot` to plot side by
side.

Visualise a matrix.

```matlab
% Plot an 8x8 matrix with a color map (using command chaining)
imagesc(magic(8)), colorbar

% Close the plot
close
```

## Control statements

```matlab
% for loop
for i = 1:10
    % double each element of matrix A
    A(i) = 2 * A(i);
end

% while loop
i = 0;
while i <= 10
    i = i + 1;
end

% if, elseif, else
if i == 5
    disp('Found a 5');
elseif i == 4
    disp('Found a 4');
else
    disp('Keep searching');
end
```

## Functions

Functions are defined in files such as `addTwoNumbers.m`. Make sure your
function is in your octave search path or current working directory.

```matlab
% File: addTwoNumbers.m

function y = addTwoNumbers(a, b)
y = a + b;
```

Pro tip: Octave can natively return multiple values.

## Vectorisation (advance topic)

By using the vectorised implementation we can take advantage of matrix multiplication provided by the linear algebra library which will be faster than implementing a summation yourself (usually with a for loop). Below is an example.

$$
h_{\theta}(x) = \sum^n_{j=0} \theta_j x_j = \theta^T x
$$

where

$$
\theta =
\begin{bmatrix}
\theta_0 \\
\theta_1 \\
\vdots \\
\theta_n
\end{bmatrix},
x =
\begin{bmatrix}
x_0 \\
x_1 \\
\vdots \\
x_n
\end{bmatrix}
$$

It is easy to see how vectorisation works.

## Further reading

- [GNU Octave Documentation](https://octave.org/doc/v5.2.0/)


Getting started with linear regression using the ordinary least squares method

You often want to find parameters that minimise some cost function. One
approach could be using gradient descent. In this article, we will be talking
about another approach called the normal equation.

## Prerequisites

- Linear algebra basics
- Vector basics

## Getting started

$$
\theta = (X^T X)^{-1} X^T y
$$

Wow! One line. Also by using this method, you don't need to do feature scaling.

### Let's verify dimensions

## Octave implementation

```matlab
pinv(X'*X)*X'*y
```

## How it works? (optional)

Please feel free to skip this section. This is quite maths heavy. Here is the
outline of the mathematical proof. For brevity, we omit the full proof and only
provide an outline.

## Cautions

- Some of you may have noticed what if $X^T X$ is non invertible (singular) -
  this is the reason we use `pinv` as it always gives us a value for $\theta$
- Need to compute $(X^T X)^{-1}$ which is slow if $n$ is very large
  since most implementations have $O(n^3)$ time complexity ($nxn$ is the dimensions)
