This is a multi-part series

• Part 1. Setting up Raspberry Pi Cluster and Installing Kubernetes
• Part 2. Setting up RBAC and Helm for Kubernetes
• Part 3. Setting up OpenFaaS

Part 1 - Setting up Raspbery Pi Cluster and Installing Kubernetes

Assemble Raspberry Pi Cluster

Recently I came upon OpenFaaS and was fascinated by the possibilities. With Kubernetes coming up on the horizon at work to migrate our workload to, it is also a good time to start digging into the mechanics of K8s. I just so happen to have a couple of Raspberry Pi 2s lying around and wanted to assemble 5 RPi to make a 1 master and 4 workers K8s cluster. I also had an extra Netgear 5 port gigabit switch and a 5 port USB charging hub that I could use for this cluster. The only thing left to order was a 5-layer acrylic RPi enclosure/shell, and 3 additional RPi (preferably model 3s).

I grabbed the acrylic enclosure on AliExpress for about ~$17. Ordering anything from AliExpress requires patience, but this one arrived in about 2 weeks.

I was able to find 3 used (still like new) condition Raspberry Pi model 3 B+ on Amazon for a total of $104.94. This was far cheaper than buying brand new. Best part about this was that each one of these came with a 32GB MicroSD Card (cha-ching!). I ended up swapping these newer B+ models with some of my older 3s and will be using the older 3s for this K8s cluster.

The Netgear 5 port gigabit switch I had was a ProSafe 5 model GS105. Purchasing this from Amazon will cost you about ~$34.99.

The 5 port USB charging hub was an EC technology Smart Charging Hub with 5V/2.4A output for each of the 5 ports. Perfect for RPi 2 or 3 power requirements! When you buy this one, make sure the one you purchase has a minimum of 5V/2.4A (~12W) power rating per port to ensure sufficient power for each of the RPi.

Assembling the acrylic enclosure was a bit tedious but self explanatory.

I used some Scotch clear mounting square double-sided adhesive to join the switch, USB charger, and RPi cluster together.

Prepare for Kubernetes Install

These are the reference sites I used to help me get up and running:

• Alex Ellis’ GH Gist: https://gist.github.com/alexellis/fdbc90de7691a1b9edb545c17da2d975
• Aaron Jones’ GH Gist: https://gist.github.com/aaronkjones/d996f1a441bc80875fd4929866ca65ad
• Edouard Lafargue’s GH Gist: https://gist.github.com/elafargue/a822458ab1fe7849eff0a47bb512546f

Starting with Alex’s instructions, I noticed that was based on Kubernetes v1.7.x which is a bit outdated. Nevertheless, I followed those instructions to try and set up a v1.11.2 K8s cluster with no luck. After some more searching, I landed on Aaron’s instructions which included v1.10.2 but again I did not have any luck installing it. I then found Edouard’s instructions using HypriotOS with v1.10.5 but again failed to get the first master node properly installed.

There were a lot of trial and error attempts with all failures. As I later found out, one of the main issue with my cluster was that I was using an RPi 2 model B as the master node. It appears to be under power and cannot handle the newer K8s v1.10.x and v1.11.x versions. Version by version, I was able to backtrack all the way down to v1.9.10 and finally was able to successfully bootstrap a master node.

Takeaway #1: Raspberry Pi 2 Model B will only work with K8s v1.9.10 or lower

Since my cluster consists of 2x RPi 2s and 3x RPi 3s, I decided to use one of the RPi 3s as a master because I wanted at least a v1.10.x K8s cluster. Again, starting with the latest version of K8s v1.11.2 with no joy. Continued to decrement the versions one by one all the way down to v1.9.10 with no luck whatsoever!

The issue was with the USB cable on this particular RPi 3. It had some sort of short that reduced the power delivery to the RPi. The telltale sign being the red PWR LED on the Pi was blinking intermittently. After replacing the USB cable, I was able to finally install K8s v1.10.6 successfully.

Takeaway #2: Ensure Raspberry Pi 3 Model B has the required 5V/2.5A (it was still happy with 2.4A from my USB hub) power else it will cause unexpected behaviors.

I tried installing v1.11.1 and v1.11.2 but did not have any luck. Turns out in order for the v1.11.x to bootstrap properly, I must use an RPi 3 Model B+. Who knew?

Takeaway #3: You’ll need the latest Raspberry Pi 3 Model B+ if you want to install K8s v1.11.x

Steps to Setting up Kubernetes Master and Worker Nodes

Kubernetes Master Node

Keeping the key takeaways in mind, I settled for K8s v1.10.6 on one of my RPi 3s which was the older Model B. I also decided to use HypriotOS because of its kickass out-of-the-box support for the latest Docker CE. These are the steps I used to bootstrap the master node:

Pre-requisites:

• I recorded all 5 of my RPi’s ethernet’s MAC addresses and has entered them into my router’s DHCP static mapping. This is especially important for the master node because you do not want the IP changing on next reboot.

1. Download HypriotOS 1.9.0

   • Since release 1.8.0, HypriotOS has included Cloud-Init capability. This allowed me to use the Hypriot’s Flash util to flash custom images to each of the RPi’s SD Cards.

2. Add Kubernetes repo to the Raspbian Aptitude list:

   $ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - &&
   echo “deb http://apt.kubernetes.io/ kubernetes-xenial main” | sudo tee /etc/apt/sources.list.d/kubernetes.list &&
   sudo apt-get update -q &&
   sudo apt-get install -qy kubeadm=1.10.6-00 kubectl=1.10.6-00 kubelet=1.10.6-00

3. Freeze the versions for kubeadm, kubectl, and kubelet so subsequent apt upgrade does not tamper with currently installed ones:

   $ sudo apt-mark hold kubeadm kubectl kubelet

4. Edit /etc/systemd/system/kubelet.service/10-kubeadm.conf and delete this line:

   Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
   

5. Initialize master node with kubeadm. We’re passing in the --pod-network-cidr because we need to install Flannel afterwards:

   $ sudo kubeadm init –pod-network-cidr=10.244.0.0/16 –kubernetes-version v1.10.6

6. If step #5 is successful, you should be given a kubeadm join command for the other K8s worker nodes.

7. Grab the admin.conf to run kubectl either on master node or locally:

   $ mkdir .kube && sudo cp /etc/kubernetes/admins.conf ~/.kube/config && sudo chown <your.user>.<your.group> ~/.kube/config

8. Test your K8s master node’s readiness:

   $ kubectl get nodes

9. Once ready, install Flannel:

   $ curl -sSL https://rawgit.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml | sed “s/amd64/arm/g” | kubectl create -f -

Kubernetes Worker Node

Pre-requisites:

• Follow steps from the Kubernetes Master Node setup steps 1 to 4.

1. Join master node. This entire join command with valid token and hash was given to you by step 5 from above.

   $ kubeadm join –token :6443 –discovery-token-ca-cert-hash sha256:

Note: if you receive this error:

[preflight] Some fatal errors occurred:
[ERROR CRI]: unable to check if the container runtime at "/var/run/dockershim.sock" is running: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...

then you can just add the option of --ignore-preflight-errors=cri to the kubeadm join command

Key Images for Verified Successful Installs

These are the successful images used for installing a particular version of Kubernetes on a master node:

Kubernetes v1.10.6

Device:   Raspberry Pi 3 Model B
Status:   Success

$ docker images
REPOSITORY                               TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-proxy-arm                v1.10.6             5108854cc889        3 weeks ago         88.1MB
k8s.gcr.io/kube-controller-manager-arm   v1.10.6             71a855d595d6        3 weeks ago         131MB
k8s.gcr.io/kube-apiserver-arm            v1.10.6             8f560fbef578        3 weeks ago         209MB
k8s.gcr.io/kube-scheduler-arm            v1.10.6             599a526309e6        3 weeks ago         44.3MB
k8s.gcr.io/etcd-arm                      3.1.12              88c32b5960ff        5 months ago        178MB
k8s.gcr.io/pause-arm                     3.1                 e11a8cbeda86        8 months ago        374kB

Kubernetes v1.11.2

Device:  Raspberry Pi 3 Model B+
Status:  Success

$ docker images
REPOSITORY                               TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-proxy-arm                v1.11.2             e73305e69b6c        10 days ago         89.1MB
k8s.gcr.io/kube-apiserver-arm            v1.11.2             279e1ebf8a61        10 days ago         170MB
k8s.gcr.io/kube-controller-manager-arm   v1.11.2             57d6be599fe9        10 days ago         138MB
k8s.gcr.io/kube-scheduler-arm            v1.11.2             68f5cf8473bd        10 days ago         50.5MB
k8s.gcr.io/etcd-arm                      3.2.18              ae02bf7047c8        4 months ago        221MB
k8s.gcr.io/pause                         3.1                 e11a8cbeda86        8 months ago        374kB

I hope the information I’ve included will help you successfully install an RPi K8s cluster without too much issues and unnecessary time spent chasing dead-end solution.

Look for Part 2 of this three part series where I’ll write a detail walkthrough on setting up Role Based Access Control (RBAC) and Helm