Setting up Raspbery Pi Cluster and Installing Kubernetes
Assemble Raspberry Pi Cluster
Recently I came upon OpenFaaS and was fascinated by the possibilities. With Kubernetes coming up on the horizon at work to migrate our workload to, it is also a good time to start digging into the mechanics of K8s. I just so happen to have a couple of Raspberry Pi 2s lying around and wanted to assemble 5 RPi to make a 1 master and 4 workers K8s cluster. I also had an extra Netgear 5 port gigabit switch and a 5 port USB charging hub that I could use for this cluster. The only thing left to order was a 5-layer acrylic RPi enclosure/shell, and 3 additional RPi (preferably model 3s).
I grabbed the acrylic enclosure on AliExpress for about ~$17. Ordering anything from AliExpress requires patience, but this one arrived in about 2 weeks.
I was able to find 3 used (still like new) condition Raspberry Pi model 3 B+ on Amazon for a total of $104.94. This was far cheaper than buying brand new. Best part about this was that each one of these came with a 32GB MicroSD Card (cha-ching!). I ended up swapping these newer B+ models with some of my older 3s and will be using the older 3s for this K8s cluster.
The Netgear 5 port gigabit switch I had was a ProSafe 5 model GS105. Purchasing this from Amazon will cost you about ~$34.99.
The 5 port USB charging hub was an EC technology Smart Charging Hub with 5V/2.4A output for each of the 5 ports. Perfect for RPi 2 or 3 power requirements! When you buy this one, make sure the one you purchase has a minimum of 5V/2.4A (~12W) power rating per port to ensure sufficient power for each of the RPi.
Assembling the acrylic enclosure was a bit tedious but self explanatory.
I used some Scotch clear mounting square double-sided adhesive to join the switch, USB charger, and RPi cluster together.
Prepare for Kubernetes Install
These are the reference sites I used to help me get up and running:
- Alex Ellis' GH Gist: https://gist.github.com/alexellis/fdbc90de7691a1b9edb545c17da2d975
- Aaron Jones' GH Gist: https://gist.github.com/aaronkjones/d996f1a441bc80875fd4929866ca65ad
- Edouard Lafargue’s GH Gist: https://gist.github.com/elafargue/a822458ab1fe7849eff0a47bb512546f
Starting with Alex’s instructions, I noticed that was based on Kubernetes v1.7.x which is a bit outdated. Nevertheless, I followed those instructions to try and set up a v1.11.2 K8s cluster with no luck. After some more searching, I landed on Aaron’s instructions which included v1.10.2 but again I did not have any luck installing it. I then found Edouard’s instructions using HypriotOS with v1.10.5 but again failed to get the first master node properly installed.
There were a lot of trial and error attempts with all failures. As I later found out, one of the main issue with my cluster was that I was using an RPi 2 model B as the master node. It appears to be under power and cannot handle the newer K8s v1.10.x and v1.11.x versions. Version by version, I was able to backtrack all the way down to v1.9.10 and finally was able to successfully bootstrap a master node.
Takeaway #1: Raspberry Pi 2 Model B will only work with K8s v1.9.10 or lower
Since my cluster consists of 2x RPi 2s and 3x RPi 3s, I decided to use one of the RPi 3s as a master because I wanted at least a v1.10.x K8s cluster. Again, starting with the latest version of K8s v1.11.2 with no joy. Continued to decrement the versions one by one all the way down to v1.9.10 with no luck whatsoever!
The issue was with the USB cable on this particular RPi 3. It had some sort of short that reduced the power delivery to the RPi. The telltale sign being the red PWR LED on the Pi was blinking intermittently. After replacing the USB cable, I was able to finally install K8s v1.10.6 successfully.
Takeaway #2: Ensure Raspberry Pi 3 Model B has the required 5V/2.5A (it was still happy with 2.4A from my USB hub) power else it will cause unexpected behaviors.
I tried installing v1.11.1 and v1.11.2 but did not have any luck. Turns out in order for the v1.11.x to bootstrap properly, I must use an RPi 3 Model B+. Who knew?
Takeaway #3: You’ll need the latest Raspberry Pi 3 Model B+ if you want to install K8s v1.11.x
Steps to Setting up Kubernetes Master and Worker Nodes
Kubernetes Master Node
Keeping the key takeaways in mind, I settled for K8s v1.10.6 on one of my RPi 3s which was the older Model B. I also decided to use HypriotOS because of its kickass out-of-the-box support for the latest Docker CE. These are the steps I used to bootstrap the master node:
- I recorded all 5 of my RPi’s ethernet’s MAC addresses and has entered them into my router’s DHCP static mapping. This is especially important for the master node because you do not want the IP changing on next reboot.
Download HypriotOS 1.9.0
Add Kubernetes repo to the Raspbian Aptitude list:
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - &&
echo “deb http://apt.kubernetes.io/ kubernetes-xenial main” | sudo tee /etc/apt/sources.list.d/kubernetes.list &&
sudo apt-get update -q &&
sudo apt-get install -qy kubeadm=1.10.6-00 kubectl=1.10.6-00 kubelet=1.10.6-00
Freeze the versions for
kubeletso subsequent apt upgrade does not tamper with currently installed ones:
$ sudo apt-mark hold kubeadm kubectl kubelet
Edit /etc/systemd/system/kubelet.service/10-kubeadm.conf and delete this line:
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Initialize master node with
kubeadm. We’re passing in the
--pod-network-cidrbecause we need to install Flannel afterwards:
$ sudo kubeadm init –pod-network-cidr=10.244.0.0/16 –kubernetes-version v1.10.6
If step #5 is successful, you should be given a
kubeadm joincommand for the other K8s worker nodes.
kubectleither on master node or locally:
$ mkdir .kube && sudo cp /etc/kubernetes/admins.conf ~/.kube/config && sudo chown
Test your K8s master node’s readiness:
$ kubectl get nodes
Once ready, install Flannel:
$ curl -sSL https://rawgit.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml | sed “s/amd64/arm/g” | kubectl create -f -
Kubernetes Worker Node
- Follow steps from the
Kubernetes Master Nodesetup steps 1 to 4.
Join master node. This entire join command with valid token and hash was given to you by step 5 from above.
$ kubeadm join –token <ip_of_master_node>:6443 –discovery-token-ca-cert-hash sha256:<long_string>
Note: if you receive this error:
[preflight] Some fatal errors occurred: [ERROR CRI]: unable to check if the container runtime at "/var/run/dockershim.sock" is running: exit status 1 [preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...
then you can just add the option of
--ignore-preflight-errors=cri to the kubeadm join command
Key Images for Verified Successful Installs
These are the successful images used for installing a particular version of Kubernetes on a master node:
Device: Raspberry Pi 3 Model B Status: Success $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE k8s.gcr.io/kube-proxy-arm v1.10.6 5108854cc889 3 weeks ago 88.1MB k8s.gcr.io/kube-controller-manager-arm v1.10.6 71a855d595d6 3 weeks ago 131MB k8s.gcr.io/kube-apiserver-arm v1.10.6 8f560fbef578 3 weeks ago 209MB k8s.gcr.io/kube-scheduler-arm v1.10.6 599a526309e6 3 weeks ago 44.3MB k8s.gcr.io/etcd-arm 3.1.12 88c32b5960ff 5 months ago 178MB k8s.gcr.io/pause-arm 3.1 e11a8cbeda86 8 months ago 374kB
Device: Raspberry Pi 3 Model B+ Status: Success $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE k8s.gcr.io/kube-proxy-arm v1.11.2 e73305e69b6c 10 days ago 89.1MB k8s.gcr.io/kube-apiserver-arm v1.11.2 279e1ebf8a61 10 days ago 170MB k8s.gcr.io/kube-controller-manager-arm v1.11.2 57d6be599fe9 10 days ago 138MB k8s.gcr.io/kube-scheduler-arm v1.11.2 68f5cf8473bd 10 days ago 50.5MB k8s.gcr.io/etcd-arm 3.2.18 ae02bf7047c8 4 months ago 221MB k8s.gcr.io/pause 3.1 e11a8cbeda86 8 months ago 374kB
I hope the information I’ve included will help you successfully install an RPi K8s cluster without too much issues and unnecessary time spent chasing dead-end solution.