Have you ever wondered how do cloud providers build their cloud ? How do circle-ci, github actions, gitlab-ci do run your tests and builds your projects in their cloud ?
Don't wonder anymore, because in this series, i'll show to build your own mini-cloud at home.
- it will require some hardware investment,
- it will give you a glimpse of hardware architecture, but not full picture (we will not cover spine/leaf nodes architecture used in cloud datacenters)
- At least two raspberry pi with 4GB each (official price 55$ each),
- One MicroSD card for each Raspberry pi (i recommend taking at least a 32GB card),
- [optional] A Raspberry pi POE+ hat for each Raspberry pi (official price 20$ each)
- [optional] A POE+ ethernet switch (or the official raspberry pi power supply if you can't afford a POE switch and want to use wifi for networking)
- [Optional] A cluster case for a clean setup.
I made one based on 4 Raspberry pi's with POE+ hat and a POE+ switch. Total Budget : 400$.
If you go with only two RPI's over WIFI, it will cost you around 150$.
We could install the official Raspberry pi OS on each card, but unfortunately, raspberry only support 32 bit images officially (their 64 bit OS is still broken from time to time), and we want to use the full power of Raspberry pi 4 which is 64 bit. See this benchmark to compare for yourself the huge difference :
So here i'll recommend using Ubuntu Server 20.04 LTS ARM64 build provided by Raspberry pi Imager.
And click on the
Before putting your SD card on your raspberry pi, you first need to make a few changes to allow you to use it in Headless mode (ie: without a screen and keyboard linked to it).
With your SD card still inserted, you should see two new disks :
Still at the root of
bootfs disk, create an empty file named
This will enable
ssh connections on your Raspberry pi.
Open a Terminal and go to the
bootfs filesystem of your SD Card, and open the
You should see a single line, append this at the end of the line :
cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
This will enable you to use kubernetes (k3s to be precise) on your cluster.
userconfig.txt and append these lines at the end of the file :
dtoverlay=rpi-poe dtparam=poe_fan_temp0=45000 dtparam=poe_fan_temp1=55000 dtparam=poe_fan_temp2=65000 dtparam=poe_fan_temp3=75000
This will enable POE+ hat driver and fan configuration to cool down your Raspberry pi.
Now go to the writable disk and edit as an admin the
etc/hosts and append at the end:
For the first SD card, i suggest naming it to recognize that's it will be the master node of your cluster. Mine is called
k3s-main. For the other ones, you can choose a predictable name like
etc/hostname and change the current name (ubuntu) to the name you just choose.
Now you can insert your SD card in your Raspberry pi's. put power on and let them boot.
Now you need to connect to each of your booted Raspberry pi's. If you have wifi and a dhcp server, you might be able to connect to them using their name with .local appended to it. For exemple :
> ssh email@example.com
When prompted for password, just use
ubuntu. it will ask you to change the default password after a successfull login.
Once the password changed, it will disconnect you.
If it does not work, don't worry. First run
ifconfig to get information about your network.
> ifconfig etho0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 inet <YOUR.LOCAL.IP> netmask 255.255.0.0 broadcast 172.19.255.255 ether <MAC:ADDRESS> txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
This should give you your local IP. replace the last two digits by
00 and run nmap :
> nmap -sn <YOUR.LOCAL.IP.ZERO>/24
This should return your your IP, plus the one of your Raspberry pi's. you should now be able to login using ssh and the raspberry IP
> ssh ubuntu@<Rasbpberry.IP>
If you have already setup a ssh key, i recommand using it to automatically connect to your Raspberry pi's without password prompt. To do this :
> ssh-copy-id -i ~/.ssh/id_rsa.pub firstname.lastname@example.org
Enter password for the last time when prompted.
On each Raspberry pi's within the SSH session, run :
> sudo apt update & sudo apt upgrade
On your main node (k3s-main for me), run the following command :
> curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
Once the script finished execution, you should be able to see your node running :
> kubectl get nodes NAME STATUS ROLES AGE VERSION k3s-main Ready control-plane,master 1m v1.21.0
save your token to allow your workers to join the main node
> cat /var/lib/rancher/k3s/server/node-token <YOUR-TOKEN>
On each of your worker nodes (k3s-worker-01, k3s-worker-02, k3s-worker-03 for me) join the main node by running the following command :
> curl -sfL https://get.k3s.io | K3S_URL=https://k3s-main.local:6443 K3S_TOKEN=<YOUR-TOKEN> sh -
Once the scripts finished execution, you should be able to see your nodes running (execute on main node):
> kubectl get nodes NAME STATUS ROLES AGE VERSION k3s-main Ready control-plane,master 15m v1.21.0 k3s-worker-01 Ready <none> 3m v1.21.0 k3s-worker-02 Ready <none> 2m v1.21.0 k3s-worker-03 Ready <none> 1m v1.21.0
So we have installed a cluster on raspberry pi's. what now ?
Next time, we'll see how to install rancher on your cluster to monitor and administrate it with a beautifull UI.