K8s cluster with OCI free-tier and Raspberry Pi4 (part 2)

This long read is a multiple part tutorial for building a Kubernetes cluster (using k3s) with 4 x OCI free-tier ARM instances and 4 x Raspberry Pi 4. Plus some applications needed for installation (Terraform and Ansible) and a lot of things installed on the cluster.
Part 2 is running a Kubernetes cluster on 4 Raspberry Pi 4 using Ansible.
GitHub repository is here

Requirements

• At least 2 Raspberry Pi. I've got 4 of them, 3 with 4GB and 1 with 8GB. Every one needs a SD Card, a power adapter plus network cables (plus an optional switch and 4 cases);
• And the same from part 1. Windows 11 with WSL2 running Ubuntu 20.04, but this will work on any Linux & Win machine and Terraform installed (tested with v1.3.7)- how to here;

My Pis

Preparing

Installing an OS on Pi is very easy. Just insert the SD card in your PC and use Imager from official website. I choose the same OS as in the OCI cluster, that is Ubuntu Server 22.04.1 64bit. In advanced settings (bottom right of Imager) pick Set Hostname and write your own. I have rpi4-1 rpi4-2 rpi4-3 and rpi4-4. Pick Enable SSH and Set username and password, this way you can connect to Pi immediately, without a monitor and keyboard for it. Then hit Write and repeat this step for every one.

From my home router I found the devices IPs. Still there you can configure Address reservation so every time they keep their IP. Did some Port forwarding too so I can access them from everywhere using DDNS from my ISP, as I don't have static IPv4. All of these settings are configured from you home router so Google how to if you're interested.

I added every PI to local machine C:\Windows\System32\drivers\etc\hosts file to be able to control them easier.

192.168.0.201 rpi4-1
192.168.0.202 rpi4-2
192.168.0.203 rpi4-3
192.168.0.204 rpi4-4

We can check if everything is ok by running some ping rpi4-1 or ssh user@rpi4-1.

Ansible

For configuration management I picked Ansible as it is agentless and not so difficult (spoiler alert, it is though). We can control all RPI4 from local machine. Install Ansible first sudo apt install ansible.
Now from your PC run the following commands (assuming you already generated ssh keys from Part 1, using ssh-keygen):

ssh-copy-id -i ~/.ssh/key.pub user@rpi4-1
ssh-copy-id -i ~/.ssh/key.pub user@rpi4-2
ssh-copy-id -i ~/.ssh/key.pub user@rpi4-3
ssh-copy-id -i ~/.ssh/key.pub user@rpi4-4

This will allow Ansible to connect to every Pi without requesting the password every time.
I had to uncomment with sudo vi /etc/ansible/ansible.cfg the line with private_key_file = ~/.ssh/key. And in /etc/ansible/hosts i added the following:

[big]
rpi4-1  ansible_connection=ssh

[small]
rpi4-2  ansible_connection=ssh
rpi4-3  ansible_connection=ssh
rpi4-4  ansible_connection=ssh

[home:children]
big
small

This will add the big rpi4 in a biggroup and the rest of workers to an small group. Plus a home group having them all. My PC user and RPIs user is the same, but if you have a different one you have to add this to the same file:

[all:vars]
remote_user = user

Now test if everything is ok with ansible home -m ping. Green is ok.
I like to keep all my systems updated to latest version, especially this one used for testing. So we'll need to create a new file update.yml and paste below block in it:

---
- hosts: home
  tasks:
    - name: Update apt repo and cache
      apt: update_cache=yes force_apt_get=yes cache_valid_time=3600
    - name: Upgrade all packages
      apt: upgrade=yes force_apt_get=yes
    - name: Check if a reboot is needed
      register: reboot_required_file
      stat: path=/var/run/reboot-required get_md5=no
    - name: Reboot the box if kernel updated
      reboot:
        msg: "Reboot initiated by Ansible for kernel updates"
        connect_timeout: 5
        reboot_timeout: 90
        pre_reboot_delay: 0
        post_reboot_delay: 30
        test_command: uptime
      when: reboot_required_file.stat.exists

This is an Ansible playbook that updates Ubuntu and then reboots the PIs. Now save it and run ansible-playbook update.yml -K -b which will ask for sudo password and run the playbook. It lasted ~10 minutes for me. On another shell you can ssh to any of the PIs and run htopto see the activity.
Now try ansible home -a "rpi-eeprom-update -a" -b -K to see if there's any firmware update for you Raspberry Pi 4.
Next step is to enable cgroups on every PI. Create a new playbook append-cmd.yml and add:

---
- hosts: home
  tasks:
  - name: Append cgroup to cmdline.txt
    lineinfile:
      path: /boot/firmware/cmdline.txt
      backrefs: yes
      regexp: "^(.*)$"
      line: '\1 cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1'
...

This will append to end of file /boot/firmware/cmdline.txt the strings cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1. Run it with ansible-playbook append-cmd.yml -K -b.
I won't add a graphical interface and won't use Wi-Fi and Bluetooth so we can steal some memory from GPU memory, Wi-Fi and BT to be available to the Kubernetes cluster. So we need to add a few lines to /boot/firmware/config.txt using a new Ansible playbook append-cfg.yml with following content:

---
- hosts: home
  tasks:
  - name: Append new lines to config.txt
    blockinfile:
      path: /boot/firmware/config.txt
      block: |
       gpu_mem=16
       dtoverlay=disable-bt
       dtoverlay=disable-wifi
...

Run it again with ansible-playbook append-cfg.yml -K -b.
Next 2 mods must be enabled with command ansible home -a "modprobe overlay" -a "modprobe br_netfilter" -K -b.
Next playbook will create 2 files and add some lines to it. Let's call it iptable.yml and add:

---
- hosts: home
  tasks:
  - name: Create a file and write in it 1.
    blockinfile:
      path: /etc/modules-load.d/k8s.conf
      block: |
        overlay
        br_netfilter
      create: yes
  - name: Create a file and write in it 2.
    blockinfile:
      path: /etc/sysctl.d/k8s.conf
      block: |
        net.bridge.bridge-nf-call-ip6tables = 1
        net.bridge.bridge-nf-call-iptables = 1
        net.ipv4.ip_forward = 1
      create: yes
...

Run it with ansible-playbook iptable.yml -K -b. After that run ansible home -a "sysctl --system" -K -b. My PIs started to be laggy so i rebooted here with ansible home -a "reboot" -K -b.
Now the last step. Installing some apps on PIs. I'm still unsure if this is needed, but I'm pretty sure it won't harm the cluster. Create a new file install.yml and add copy this:

---
- hosts: home
  tasks:
  - name: Install some packages
    apt:
      name:
        - curl
        - gnupg2
        - software-properties-common
        - apt-transport-https
        - ca-certificates
        - linux-modules-extra-raspi
...

Now run it with ansible-playbook install.yml -K -b.

That's all folks. For this part. If you are here only for the Raspberry part now you should install k3s very easy, just follow the steps from official documentation.

References

One of my old k3s install on Raspbbery PI article on LinkedIn - here - inspired from braindose.blog;
Official Ansible documentation - here;
ChatGPT helped me a lot of time, use it here.