K8s cluster with OCI free-tier and Raspberry Pi4 (part 1)

This long read is a multiple part tutorial for building a Kubernetes cluster (using k3s) with 4 x OCI free-tier ARM instances and 4 x Raspberry Pi 4. Plus some applications needed for installation (Terraform and Ansible) and a lot of things installed on the cluster.
Part 1 is running a Kubernetes cluster on OCI free-tier resources using Terraform.
GitHub repository is here

Requirements

• Obvious, an OCI account, get it from here - oracle.com/cloud. If you already have an account, be careful not to have any resources provisioned already (even for other users, or compartments), this tutorial will use all free-tier ones. Also be extra careful to pick a region not so popular, as it may have no resources available. Pick a region with enough ARM instances available. If during final steps, terraform is stuck, you can check in OCI > Compute > Instance Pools > select your own > Work requests , if there is Failure _and in that log file there's an error _Out of host capacity, then you must wait, even days until resources are freed. You can run a script from here which will try to create instances until there's something available. When that happens, go fast to your OCI, delete all that was created and then run the terraform scripts;
• I used Windows 11 with WSL2 running Ubuntu 20.04, but this will work on any Linux machine;
• Terraform installed (tested with v1.3.7 - and OCI provider v4.105)- how to here;

Preparing

(following official guidelines from Oracle)

For safety we should use a separate compartment and user for our OCI configuration, not root ones. Now is a good time to create a new notes file and add some values there, you will need them later. Mostly you will add 3 things for each value (user + user_name_you_created + it's_OCID; group ... etc.).
Go to Identity & Security > Compartments and click on Create Compartment. Open it and copy the OCID to your notes file. Then in Identity & Security > Users click on Create User. Open it and copy the OCID to your notes file.
Then in Identity & Security > Groups click on Create Group. The same as above with the OCID. Here click on Add User to Group and add the newly created user.
In Identity & Security > Policies click on Create Policy, Show manual editor and add the following

allow group group_you_created to read all-resources in <compartment compartment_you_created>
allow group group_you_created to manage virtual-network-family  in compartment <compartment_you_created>
allow group group_you_created to manage instance-family  in compartment <compartment_you_created>
allow group group_you_created to manage compute-management-family  in compartment <compartment_you_created>
allow group group_you_created to manage volume-family  in compartment <compartment_you_created>
allow group group_you_created to manage load-balancers  in compartment <compartment_you_created>
allow group group_you_created to manage network-load-balancers  in compartment <compartment_you_created>
allow group group_you_created to manage dynamic-groups in compartment <compartment_you_created>
allow group group_you_created to manage policies in compartment <compartment_you_created>
allow group group_you_created to manage dynamic-groups in tenancy

Then you need access to OCI from your machine. So, create a new folder in HOME directory
mkdir ~/.oci
Generate a private key there
openssl genrsa -out ~/.oci/key.pem 2048
Change permissions for it
chmod 600 ~/.oci/key.pem
Then generate you're public key
openssl rsa -pubout -in ~/.oci/key.pem -out $HOME/.oci/key_public.pem
And then copy that public key, everything inside that file
cat ~/.oci/key_public.pem
This key has to be added to your OCI new user. Go to OCI > Identity & Security > Users select the new user and open API Keys , click on Add API Key, select Paste Public Key, and there paste all your copied key.
After you've done that you need to copy to notes the fingerprint too. Save the path to the private key too.
*note: Use ~ and not $HOME, that's the only way it worked for me.

To your notes file copy the following too:

• Tenancy OCID. Click on your avatar (from top-right), and select Tenancy.
• Region. In the top right there is the name of your region too. Now find it here and copy it's identifier (ex. eu-paris-1).
• The path to the private key. In our case - ~/.oci/key.pem

Now create a new folder to test if terraform is ok and linked with OCI. In that folder create a file main.tf and add this:

terraform {
  required_providers {
    oci = {
      source  = "oracle/oci"
      version = "4.105.0"
    }
  }
}


# Configure the OCI provider with an API Key

provider "oci" {
  tenancy_ocid     = "ocid1.tenancy.oc1..aaaaaaYOURTENANCY3uzx4a"
  user_ocid        = "ocid1.user.oc1..aaaaaaYOURUSER4s5ga"
  private_key_path = "~/.oci/oci.pem"
  fingerprint      = "2a:d8:YOURFINGERPRINT:a1:cd:06"
  region           = "eu-YOURREGION"
}

#Get a list of Availability Domains
data "oci_identity_availability_domains" "ads" {
  compartment_id = "ocid1.tenancy.oc1..aaaaaaYOURTENANCYuzx4a"
}

#Output the result
output "all-availability-domains-in-your-compartment" {
  value = data.oci_identity_availability_domains.ads.availability_domains
}

Now you should run terraform init to download the OCI provider and then terraform plan to see what will happen and terraform apply to receive final results. This small demo configuration file from above should return the name of the availability domains in that regions. If you receive something like "name" = "pmkj:EU-YOURREGION-1-AD-1" and no errors then everything is ok until now. This file can be deleted now.

Provisioning

You will need to add some new values to the notes file:

• From inside OCI, on top right corner, click on Developer Tools > Cloud Shell and there write oci iam availability-domain list. Save the name, not id (if more than 1, pick one). This is your availability_domain variable;
• Again from this console type oci compute image list --compartment-id <YOUR-COMPARTMENT> --operating-system "Canonical Ubuntu" --operating-system-version "22.04 Minimal aarch64" --shape "VM.Standard.A1.Flex", to find the OS Image ID. Probably the first result has the latest build, in my case was Canonical-Ubuntu-22.04-Minimal-aarch64-2022.11.05-0. From here save the id. This is your os_image_id variable;
• Now just google "my ip" and you will find your Public IP. Save it in CIDR format, ex. 111.222.111.99/32. This is your my_public_ip_cidr variable. I use a cheap VPS just to have a static IP. If you don't have a static IPv4 from your ISP, I don't know a quick solution for you, maybe someone can comment one. You can setup DDNS, but that can't be used in Security List afaik. Only solution every time your IP changes, go to VCN > Security List and modify the Ingress rule with the new IP;
• Your public_key_path is your public SSH keys. If you don't have any, quickly generate them with ssh-keygen. You should have one now in ~/.ssh/key.pub (I copied the private key, using scp to the VPS, so I can connect to OCI from local machine and from VPS);
• Last is your email address that will be used to install a certification manager. That will be your certmanager_email_address variable. I didn't setup one, as this is just a personal project for testing.

After you've cloned the repo, go to oci/terraform.tfvars and edit all values with the ones from your notes file.
This build uses the great terraform configuration files from this repo of garutilorenzo (using version 2.2; if you have errors running all of this, you should check what changed in this repo since v2.2, or 01.02.23). You can read here if you want to customize your configuration and edit the main.tf file. This is the diagram that garutilorenzo made and how your deployment will look like (this tutorial is without Longhorn and ArgoCD, with 1 server nodes + 3 worker nodes and with ingress controller set as Traefik):

*note - I've got some problems with clock of WSL2 not being synced to Windows clock. And provisioning didn't worked so if you receive clock errors too, verify your time with datecommand, if out of sync just run sudo hwclock -s or sudo ntpdate time.windows.com.
Now just run terraform plan and then terraform apply. If everything was ok you should have your resources created.

When the script finishes save the outputs (or you can find the values in OCI):

Outputs:
k3s_servers_ips = [
  "152.x.x.115",
]
k3s_workers_ips = [
  "140.x.x.158",
  "140.x.x.226",
]
public_lb_ip = tolist([
  {
    "ip_address" = "140.x.x.159"
    "ip_version" = "IPV4"
    "is_public" = true
    "reserved_ip" = tolist([])
  },
  {
    "ip_address" = "10.0.1.96"
    "ip_version" = "IPV4"
    "is_public" = false
    "reserved_ip" = tolist([])

Now you can connect to any worker or server IP using ssh -i ~/.ssh/key ubuntu@152.x.x.115. Connect to server IP and write sudo kubectl get nodes to check all nodes.

That's all for now.

References

Official OCI provider documentation from Terraform - here.
Official OCI Oracle documentation with Tutorials - here and Guides - here.
Great GitHub repo of garutilorenzo - here. There are a few others who can help you with k8s on OCI too 1 with repo, 2, 3.