Install & Manage Amazon EKS Add-ons with Terraform

What is an Add-on in Amazon EKS ?

An add-on is software that provides supporting operational capabilities to Kubernetes applications, but is not specific to the application.
This includes software like observability agents or Kubernetes drivers that allow the cluster to interact with underlying AWS resources for networking, compute, and storage.

Reference: https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html

Amazon EKS Add-ons

Amazon VPC CNI
The Amazon VPC CNI add-on for Kubernetes is the networking plugin for pod networking in Amazon EKS clusters. The plugin is responsible for allocating VPC IP addresses to Kubernetes nodes and configuring the necessary networking for pods on each node.

Reference: https://docs.aws.amazon.com/eks/latest/userguide/cni-iam-role.html

CoreDNS
CoreDNS is a flexible, extensible DNS server that can serve as the Kubernetes cluster DNS. When you launch an Amazon EKS cluster with at least one node, two replicas of the CoreDNS image are deployed by default, regardless of the number of nodes deployed in your cluster. The CoreDNS pods provide name resolution for all pods in the cluster.

Reference: https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html

kube-proxy
Kube-proxy maintains network rules on each Amazon EC2 node. It enables network communication to your pods.

Reference: https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html

Amazon EBS CSI
The Amazon Elastic Block Store (Amazon EBS) Container Storage Interface (CSI) driver allows Amazon Elastic Kubernetes Service (Amazon EKS) clusters to manage the lifecycle of Amazon EBS volumes for persistent volumes.

Reference: https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html
https://docs.aws.amazon.com/eks/latest/userguide/managing-ebs-csi.html

Amazon EKS automatically installs self-managed add-ons such as the Amazon VPC CNI, kube-proxy, and CoreDNS for every cluster, but you can change the default configuration of the add-ons and update them when desired.

Amazon EBS CSI Driver is now available as an Add-on. This Add-on is in preview version with some limitations and inconsistencies. For this reason, its use in Production is not recommended.

This post will show the step by step how to install the Add-ons with terraform

Prerequisites

• Amazon EKS cluster running Kubernetes version 1.18 and later

To create an EKS cluster you can check this post https://dev.to/aws-builders/creating-an-eks-cluster-and-node-group-with-terraform-1lf6

• AWS CLI installed

https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

• An IAM role with the AmazonEKS_CNI_Policy attached to it

STEPS

Step 01 - Get the version of each Add-on

To know the all version of each Add-on we can use AWS CLI to describe all information about the Add-ons

aws eks describe-addon-versions > addons.json

This command allows to describe all add-on versions and for more confort we can save the output in a JSON file

The resulting JSON file will be like this

{
    "addons": [
        {
            "addonName": "kube-proxy",
            "type": "networking",
            "addonVersions": [
                {
                    "addonVersion": "v1.21.2-eksbuild.2",
                    "architecture": [
                        "amd64",
                        "arm64"
                    ],
                    "compatibilities": [
                        {
                            "clusterVersion": "1.21",
                            "platformVersions": [
                                "*"
                            ],
                            "defaultVersion": true
                        }
                    ]
                },
...

This JSON file contains all information about the EKS cluster Add-ons, like name, type, compatibilities, etc.

Take a note of each add-on version to specify them in the terraform code, according to the cluster version

Step 02 - Create the terraform code

First, We can create a variable of the type list(object()) to specify all names and versions of each Add-ons, based on the information that we get from the JSON file obtained in the last step

variable "addons" {
  type = list(object({
    name    = string
    version = string
  }))

  default = [
    {
      name    = "kube-proxy"
      version = "v1.21.2-eksbuild.2"
    },
    {
      name    = "vpc-cni"
      version = "v1.10.1-eksbuild.1"
    },
    {
      name    = "coredns"
      version = "v1.8.4-eksbuild.1"
    },
    {
      name    = "aws-ebs-csi-driver"
      version = "v1.4.0-eksbuild.preview"
    }
  ]
}

The AWS Provider for Terraform has a specific resource to install and manage the Add-ons.

Using a for_each statement we can iterate all information in the addons variable

resource "aws_eks_addon" "addons" {
  for_each          = { for addon in var.addons : addon.name => addon }
  cluster_name      = aws_eks_cluster.eks-cluster.id
  addon_name        = each.value.name
  addon_version     = each.value.version
  resolve_conflicts = "OVERWRITE"
}

Step 03 - Apply the terraform code

To apply the new resource we can use

terraform apply -auto-approve

Once the installation is finished, we can check the add-ons using the UI

The installation was successful so we can see the Status of each Add-on is Active

Another way to check that everything is can be using kubectl

kubectl get po -n kube-system

With this command we can check if the pods of each Add-on are running