Job Céspedes Ortiz

Posted on Nov 13, 2023 • Originally published at jobcespedes.dev

Building a K3s Cluster with Fedora CoreOS on Orange Pi 5 Plus: Lessons from My Unsuccessful Attempt

#orangepi #fedora #kubernetes #krestomatio

This blog post summarizes my unsuccessful attempt to build a K3s Cluster on the Orange Pi 5 Plus (opi5+) using Fedora CoreOS (FCOS). It also outlines the steps taken before arriving at that conclusion and presents a list of possible alternatives to achieve a similar result.

Why Fedora CoreOS?

The main reasons behind selecting FCOS as the Operating System (OS) were:

Automatic Updates: Fedora CoreOS follows an automated update model, and reboots can also be orchestrated using Zincati.
Immutable OS: The operating system is designed to be immutable, meaning that changes are made by replacing the entire OS image rather than modifying the existing system. This helps improve stability and reproducibility.
Container-Optimized: It is optimized for running containers and container orchestration platforms like Kubernetes. This makes it well-suited for modern cloud-native applications.
RPM-OSTree: Fedora CoreOS uses RPM-OSTree, which allows for atomic updates and rollbacks, making it easier to maintain system consistency and recover from issues.
Security: Frequent updates and immutability contribute to a more secure system. Security patches are quickly applied to minimize vulnerabilities.
Community, Ecosystem, and Open Source: Being part of the Fedora Project and Open Source, Fedora CoreOS benefits from a vibrant community and a rich ecosystem of tools and applications.
Don't start from scratch: As part of Krestomatio tools for automating infrastructure, a terraform module was developed to help the deployment of a K3s Cluster using FCOS. The module outputs ignitions (ign) files required, just by setting some variables.

Warning

There are a couple of things you should be aware of:

This is a laboratory environment.
Ensure that you use the correct official images and URL for your device model, where aplicable. For instance, the opi5+ images and urls differs from the opi5 ones.
Some steps involve commands that will erase data.

Assumptions

The following assumptions have been made:

Device is Orange Pi 5+ version.
There is another Fedora-based laptop/PC for preparations.

Preparations

Items

The items used for this task are:

1x SD Card with at least 4GB
1x USB Drive with at least 4GB
1x Orange Pi5 Plus 16G + 256G EMMC

SD Card

For the SD card:

Check SD card requirements in opi5+ manual
Install OS image, following the opi5 official docs for burning it to the SD.

USB Drive

In the USB drive;

Copy EDK2 latest release, naming it edk2-firmware.img
Copy FCOS image, generate it with the step below, naming it fcos.img

FCOS image

To prepare the FCOS image:

Create an ignition file and store it in /tmp/config.ign
Create FCOS image

# Create FCOS image
STREAM=next # CHANGEME: `stable` or `testing` or `next`
FCOS_IMG=fcos.img
IGN_FILE=/tmp/config.ign # CHANGEME
CONTAINER_BUILDER=podman # CHANGEME: `podman` or `docker`
sudo losetup -D
rm -f "${PWD}/${FCOS_IMG}"
truncate -s 3GB "${PWD}/${FCOS_IMG}"
sudo losetup -f -P "${PWD}/${FCOS_IMG}"
sudo $CONTAINER_BUILDER run --pull=always --privileged --rm \
  -v /dev:/dev -v /run/udev:/run/udev -v $IGN_FILE:$IGN_FILE \
  quay.io/coreos/coreos-installer:release \
  install --firstboot-args=console=tty0 \
  -p "metal" -a aarch64 -s $STREAM -i $IGN_FILE /dev/loop0
sudo sync
sudo losetup -D

First Boot

Once preparations are done, continue:

Insert and boot from SD Card
Insert USB drive and mount it to /mnt
Copy EDK2 to SPI, following steps below
Copy FCOS image to eMMC, following steps below
Power off
Remove SD Card
Boot from eMMC

EDK2 to SPI

To install EDK2 to SPI:

Erase SPI
Copy EDK2 to SPI

# erase SPI
sudo dd if=/dev/zero of=/dev/mtdblock0
sudo sync
# copy EDK2 to SPI
sudo dd if=edk2-firmware.img of=/dev/mtdblock0
sudo sync

FCOS image to eMMC

Identify eMMC device
Erase eMMC device
Copy FCOS image to eMMC

# identify emmc device. Ex: /dev/mmcblk1
ls /dev/mmcblk*boot0 | cut -c1-12
# erase
sudo dd bs=1M if=/dev/zero of=/dev/mmcblk1 count=1000 status=progress
sudo sync
# copy
sudo dd bs=1M if=fcos.img of=/dev/mmcblk1 count=1000 status=progress
sudo sync

Result

Remember to power off and remove the SD card. Then test booting from FCOS image in eMMC on the opi5+.

The generic FCOS image does not work out of the box. It throws the error Timed out wating for device for boot and root devices and then enters emergency mode. Attempts to run using F39 and F40 were made, with the same result though. I suspect that the eMMC is not being recognized. I did not proceed beyond this point. Instead, I began considering alternatives to achieve my goal of building a K3s cluster.

Alternatives

After this result, there are at least four (4) alternatives I am considering to build a K3s Cluster using opi5+ devices. Those are:

Use official or third party images + K3s
Use Ubuntu + Libvirt + Fedora CoreOS VM + K3s
Build a custom Fedora CoreOS, with Orange Pi linux Kernel + K3s
Wait for full RK3588 Mainline Kernel support

DEV Community