9 Ways to Spin Up an EKS Cluster - Way 3 - eksctl

In my previous post I showed how to spin up an EKS cluster with pure shell and AWS CLI. (All the links to other posts in this series will be here)

This used to be the easiest way of getting to a cluster without leaving your terminal. But pretty early in EKS history (2017) some smart folks from a company named Weaveworks(RIP) realized it was too cumbersome to do this using the aws cli subcommand and that EKS is complex enough to deserve a command-line client of its own. That's how eksctl was born.

A few months ago Weaveworks (who brought us a plethora of great OSS tools like Flux, Flagger and Weave) was shut down. But AWS announced full support for eksctl in 2019 - so eksctl is now the de-facto standard EKS CLI tool.

The great thing about eksctl is that it allows one to create and manage clusters not only using one-off commands with arguments but also with YAML configuration files - in a true and familiar IaC way.

We'll check out both options but first let's install eksctl and generate an SSH key so we can connect to the nodes in the clusters we create if needed. Please note - I'm not endorsing SSH connections to your EKS nodes. Do avoid this if possible - so as not to cause inadvertent configuration drift. But sometimes we still need this for troubleshooting, especially in training environments. So let's have the SSH key handy.

Install eksctl

If you're on Linux - here are the official instructions:

# for ARM systems, set ARCH to: `arm64`, `armv6` or `armv7`
ARCH=amd64
PLATFORM=$(uname -s)_$ARCH

curl -sLO "https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_$PLATFORM.tar.gz"

# (Optional) Verify checksum
curl -sL "https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_checksums.txt" | grep $PLATFORM | sha256sum --check

tar -xzf eksctl_$PLATFORM.tar.gz -C /tmp && rm eksctl_$PLATFORM.tar.gz

sudo mv /tmp/eksctl /usr/local/bin

Please note this doesn't install such eksctl prerequisites as kubectl and aws-iam-authenticator.

And if, like me - you're on a Mac - definitely use brew as it takes care of all dependencies. (even though the official eksctl docs don't recommend it)

brew tap weaveworks/tap
brew install weaveworks/tap/eksctl

And now - let's generate that ssh key:

ssh-keygen  -f ./id_rsa -N ''

This will create an id_rsa and id_rsa.pub in your current directory. Make sure to run the following eksctl commands from the same directory and it will pick up this key by default.

Sidenote - the VPC

If you've read the previous post in this series (where we created an EKS cluster using the AWS CLI), you'd notice that creating the VPC was a separate step. The added value of eksctl is it takes care of most dependencies and add-ons for us without the need of running additional commands. The same is true for VPC creation. A new VPC with default subnet configuration is created for us each time we spin up a new cluster, unless we specifically define we want to re-use an existing VPC.

1. Create an EKS cluster - eksctl with arguments

The most straightforward way of creating an EKS cluster with eksctl is providing all the arguments on the command-line and letting the tool take care of the defaults. This approach, while limited and not repeatable enough can definitely give us a cluster.

The command I provide here defines quite a number of settings I personally find important even for small toy clusters I spin up for fun and games. But eksctl can do its job even with less stuff defined. Look in the official "Getting Started" docs if you want just the bare bones.

So here's what I decided to use:

# First - define the environment. 
export CLUSTER_NAME=way3
export AWS_REGION=eu-central-1
export K8S_VERSION=1.30
export NODE_TYPE=t2.medium
export MIN_NODES=1
export MAX_NODES=3

I'm starting out with small nodes and already preparing the cluster for auto-scaling with min and max nodes definitions. It's important to note that eksctl allows us to enable the IAM policy for ASG acces and define the auto-scaling range. But it doesn't take care of installing cluster-autoscaler. We'd need to do that separately. If we wanted to... On the other hand - these days it makes total sense to start out with Karpenter. For which eksctl does provide support, but not on the command line. whcih means we'll see how to configure Karpenter in the next section.

And now - time to spin up the cluster:

eksctl create cluster --name $CLUSTER_NAME \
                      --region $AWS_REGION \
                      --with-oidc --version $K8S_VERSION \
                      --nodegroup-name ng-$CLUSTER_NAME-1 \
                      --node-type t2.medium \
                      --nodes 1 --nodes-min 1 --nodes-max 3 \ 
                      --spot \
                      --ssh-access \
                      --asg-access \
                      --external-dns-access \
                      --full-ecr-access \
                      --alb-ingress-access

This command gives us a full-featured cluster with IAM policies for ECR access (--full-ecr-access), external dns controller (--external-dns-access) , ALB ingress controller (--alb-ingress-access), OIDC support and more. It also runs its nodes on spot instances for cost optimization. Which is totally fine for a toy cluster but may be not appropriate if the application you're planning to deploy isn't disruption-tolerant.

From the command output we learn that in the background our command is converted into a couple of CloudFormation stacks:

2024-06-27 12:51:47 [ℹ]  will create a CloudFormation stack for cluster itself and 0 nodegroup stack(s)
2024-06-27 12:51:47 [ℹ]  will create a CloudFormation stack for cluster itself and 1 managed nodegroup stack(s)

After about 15 minutes (depending on the weather and the region you've decided to use) CloudFormation returns and we can access our cluster:

kubectl get node
NAME                                             STATUS   ROLES    AGE   VERSION
ip-192-168-56-76.eu-central-1.compute.internal   Ready    <none>   35m   v1.29.3-eks-ae9a62a

Note that the new cluster context is added to your kubeconfig automatically.
If you want to update the kubeconfig at a later time you can use:

eksctl utils write-kubeconfig -c $CLUSTER_NAME -r $AWS_REGION

But, as we already said - the CLI approach is limited. To do real IaC we want to put the cluster definitions in a YAML config file. This gives us a lot more capabilities, and allows to commit the config file to source control for further collaboration, change tracking and automation.

But first - let's remove the cluster we just created:

eksctl delete cluster --region=$AWS_REGION --name=$CLUSTER_NAME

2. Create an EKS cluster - eksctl with a config file.

The config file I provide here gives us everything we defined at the command line and more. As mentioned - it also allows us to install Karpenter in the same eksctl execution - thus giving us an industry-standard auto-scaling EKS cluster with just-in-time node provisioning. You can grab this file in Github too.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: way3
  region: eu-central-1
  version: "1.30"
  tags:
    karpenter.sh/discovery: way3
iam:
  withOIDC: true

managedNodeGroups:
  - name: ng-way3-1
    labels: { role: worker }
    instanceType: t2.medium
    desiredCapacity: 2
    minSize: 1
    maxSize: 3
    tags:
      nodegrouprole: way3
    volumeSize: 20
    iam:
      withAddonPolicies:
        externalDNS: true
        certManager: true
        awsLoadBalancerController: true
        albIngress: true
        ebs: true
        efs: true
        imageBuilder: true
        cloudWatch: true
    ssh:
      allow: true # will use ~/.ssh/id_rsa.pub as the default ssh key

karpenter:
  version: '0.37.0'
  createServiceAccount: true
  withSpotInterruptionQueue: true

An attentive eye will also notice I've also defined some additional stuff such as CloudWatch logging of the control plane, EBS and EFS access. Consider removing these lines if you don't need them.
Also you'll notice that not only it installs Karpenter, it also takes care of setting up the SpotInterruptionQueue, which allows Karpenter to replace spot instances before they die.
And there are many additional options available.
So yes - this is a very scalable approach, which takes care of more or less everything one might need in an EKS cluster.

Execute this plan with:

eksctl create cluster -f cluster.yaml

This again creates a CloudFormation execution that, granted we have all the necessary permissions, should complete successfully.

Let's check that Karpenter got installed:

kubectl get pod -A
NAMESPACE     NAME                         READY   STATUS    RESTARTS   AGE
karpenter     karpenter-79db484bbf-flzzq   1/1     Running   0          32s
karpenter     karpenter-79db484bbf-nfhsp   1/1     Running   0          32s
kube-system   aws-node-8h4ln               2/2     Running   0          17m
kube-system   aws-node-vq8wj               2/2     Running   0          18m
kube-system   coredns-6f6d89bcc9-qx497     1/1     Running   0          24m
kube-system   coredns-6f6d89bcc9-wwjtp     1/1     Running   0          24m
kube-system   kube-proxy-8mnd2             1/1     Running   0          18m
kube-system   kube-proxy-c5zkp             1/1     Running   0          17m

Yup, here it is!

The upside of using the config file is of course the ability to manage stuff in a somewhat idempotent way. So for example if we want to change our node group config - we can update the following lines:

- name: ng-1
    labels: { role: worker }
    instanceType: t2.medium
    desiredCapacity: 1
    minSize: 1
    maxSize: 5

and then run eksctl update nodegroup -f cluster.yaml - this will update our NodeGroup autoscaling range.

And of course eksctl provides us with a plethora of addtional commands that come very handy for ongoing management of EKS clusters:

eksctl -h
The official CLI for Amazon EKS

Usage: eksctl [command] [flags]

Commands:
  eksctl anywhere                        EKS anywhere
  eksctl associate                       Associate resources with a cluster
  eksctl completion                      Generates shell completion scripts for bash, zsh or fish
  eksctl create                          Create resource(s)
  eksctl delete                          Delete resource(s)
  eksctl deregister                      Deregister a non-EKS cluster
  eksctl disassociate                    Disassociate resources from a cluster
  eksctl drain                           Drain resource(s)
  eksctl enable                          Enable features in a cluster
  eksctl get                             Get resource(s)
  eksctl help                            Help about any command
  eksctl info                            Output the version of eksctl, kubectl and OS info
  eksctl register                        Register a non-EKS cluster
  eksctl scale                           Scale resources(s)
  eksctl set                             Set values
  eksctl unset                           Unset values
  eksctl update                          Update resource(s)
  eksctl upgrade                         Upgrade resource(s)
  eksctl utils                           Various utils
  eksctl version                         Output the version of eksctl

All in all - eksctl is the go to tool for EKS management if you haven't already standardized your cloud platform on another IaC solution such as Terraform, Pulumi, CDK or others which we'll look into in the folowing posts.

Thanks for reading and may your clusters be lean!

P.S. now you got a cluster - why not start managing its cost and performance for free with PerfectScale - the leading Kubernetes cost optimization solution?

Join now to build clusters you can be proud of: https://perfectscale.io.