Ultimate EKS Baseline Cluster: Part 1 - Provision EKS

The AWS managed Kubernetes service, EKS (Elastic Kubernetes Service), has the highest level of complexity amongst cloud offerings.

Besides having to build out the networking, routing, security, and worker nodes separately from the managed master nodes, there’s no longer any bundled support for storage starting from 1.23, and the existing support for external load balancers is limited.

This tutorial, presented in separate parts, will demonstrate how to build out a robust Kubernetes cluster with just a few commands.

1. EKS Cluster with support for OIDC
2. Add Storage support: persistent volumes using EBS
3. Add load balancer support: layer 4 (NLB) and layer 7 (ALB)
4. Add traffic security support: network policies using Calico
5. Demo application with Dgraph putting it all together

In order to secure services that need access to cloud resources, an OIDC provider through IAM Roles for Service Account facility will be used.

0. Prerequisites

These are are some prerequisites and initial steps needed to get started before provisioning a Kubernetes cluster and installing add-ons.

0.1 Knowledge: Systems

Basic concepts of systems, such as Linux and the shell (redirection, pipes, process substitution, command substitution, environment variables), as well as virtualization and containers are useful. The concept of a service (daemon) is important.

0.2 Knowledge: Networking

This article requires some basic understanding of networking with TCP/IP and the OSI model, specifically the Transport Layer 4 and Application Layer 7 for HTTP. This article covers using load balancing and reverse proxy.

In Kubernetes, familiarity with service types: ClusterIP, NodePort, LoadBalancer, ExternalName and the ingress resource are important.

Exposure to other types of Kubernetes resource objects used in this guide are helpful: persistentvolumeclaims (pvc), storageclass (sc), pods, deployments, statefulsets (sts), configmaps, serviceaccount (sa) and networkpolicies.

0.4 Tools

These are the tools used in this article series:

• AWS CLI [aws] is a tool that interacts with AWS.
• kubectl client [kubectl] a the tool that can interact with the Kubernetes cluster. This can be installed using adsf tool.
• helm [helm] is a tool that can install Kubernetes applications that are packaged as helm charts.
• eksctl [eksctl] is the tool that can provision EKS cluster as well as supporting VPC network infrastructure.
• POSIX Shell [sh] such as bash[bash] or zsh [zsh] are used to run the commands.

These tools are highly recommended:

• adsf [adsf] is a tool that installs versions of popular tools like kubectl.
• jq [jq] is a tool to query and print JSON data
• GNU Grep [grep] supports extracting string patterns using extended Regex and PCRE.

0.5 AWS Setup

Before getting started on EKS, you will need to set up billing to an AWS account (there’s a free tier), and then configure a profile that has provides access to an IAM User identity. See Setting up the AWS CLI for more information on configuring a profile.

After setup, you can test access with the following below:

export AWS_PROFILE="<your-profile-goes-here>"
aws sts get-caller-identity

This should show something like the following with values appropriate to your environment, e.g. example output to IAM user named kwisatzhaderach:

0.6 Kubernetes Client Setup

If you use asdf to install kubectl, you can get the latest version with the following:

# install kubectl plugin for asdf
asdf plugin-add kubectl \
  https://github.com/asdf-community/asdf-kubectl.git
asdf install kubectl latest

# fetch latest kubectl 
asdf install kubectl latest
asdf global kubectl latest

# test results of latest kubectl 
kubectl version --short --client 2> /dev/null

This should show something like:

Client Version: v1.27.1
Kustomize Version: v5.0.1

Also, create directory to store Kubernetes configurations that will be used by the KUBECONFIG env variable:

mkdir -p $HOME/.kube

0.7 Setup Environment Variables

These environment variables will be used throughout this guide. If opening up a new browser tab, make sure to set the environment variables accordingly.

# variables used to create EKS
export AWS_PROFILE="my-aws-profile" # CHANGEME
export EKS_CLUSTER_NAME="my-unique-cluster-name" # CHANGEME
export EKS_REGION="us-west-2"
export EKS_VERSION="1.26"
# KUBECONFIG variable
export KUBECONFIG=$HOME/.kube/$EKS_REGION.$EKS_CLUSTER_NAME.yaml

# account id
export ACCOUNT_ID=$(aws sts get-caller-identity \
  --query "Account" \
  --output text
)

1. Provision an EKS cluster

After prerequisite tools are installed and setup, we can start provisioning cloud resources and deploy components to Kubernetes.

1.1. Provision the EKS Cluster

The cluster can be brought up with the following command:

eksctl create cluster \
  --version $EKS_VERSION \
  --region $EKS_REGION \
  --name $EKS_CLUSTER_NAME \
  --nodes 3

Also, check the status of the worker nodes and applications running on Kubernetes.

kubectl get nodes
kubectl get all --all-namespaces

This should show something like the following.

1.2. Install the Kubernetes client

Once this finished in about 20 minutes to provision both VPC and EKS, install a kubectl version that matches the Kubernetes server version:

# fetch exact version of Kubernetes server (Requires GNU Grep)
VER=$(kubectl version --short 2> /dev/null \
  | grep Server \
  | grep -oP '(\d{1,2}\.){2}\d{1,2}'
)

# setup kubectl tool
asdf install kubectl $VER
asdf global kubectl $VER

⚠️ NOTE: The above command requires GNU grep. If you have Homebrew, you can run brew install grep. Windows can get this with MSYS2 or git-bash.

1.3 Add OIDC Provider Support

The EKS cluster has an OpenID Connect (OIDC) issuer URL associated with it. To use IRSA (IAM roles for service accounts), an IAM OIDC provider must exist for the cluster’s OIDC issuer URL.

You can set this up with the following command:

eksctl utils associate-iam-oidc-provider \
  --cluster $EKS_CLUSTER_NAME \
  --region $EKS_REGION \
  --approve

You can verify the OIDC provider is added with the following:

OIDC_ID=$(aws eks describe-cluster \
  --name $EKS_CLUSTER_NAME \
  --region $EKS_REGION \
  --query "cluster.identity.oidc.issuer" \
  --output text \
  | cut -d '/' -f 5
)

aws iam list-open-id-connect-providers \
  | grep $OIDC_ID \
  | cut -d '"' -f4 \
  | cut -d '/' -f4

Resources

AWS Cloud Resources

These are some of the AWS resource objects used in this guide.

• Elastic Kubernetes Service (EKS)
• Virtual Compute Cloud (VPC)
• Elastic Computer Cloud (EC2)
• AutoScale Group (AGS)
• Managed Node Groups

Conclusion

This first article shows have to get a modern Kubernetes cluster working with support for OIDC that will allow restricted access to cloud resources from Kubernetes applications using concept of least privilege

This article uses eksctl tool to provision the necessary cloud resources to bring up EKS. The eksctl tool will use Cloud Formation stacks to deploy the necessary cloud resources (VPC, EC2, SGs, etc) that make up the full solution.

I am sure there are many wondering, why not use something like Terraform for provisioning?

That is actually a good idea except that this will be more involved because we'll have to deploy all of the components required to make a functional EKS cluster: VPC, internet gateways, nat groups, subnet, routing tables, auto-scale groups, security groups, etc. in addition to the master control plane (EKS).

In the future, I would like to make a similar series with Terraform to setup the base EKS cluster, and then use Terraform with templatefile or Helmfile to manage installing helm charts and kubernetes manifests.

The ultimate goal of this series is to quickly get a robust and secure Kubernetes cluster with necessary features like storage and networking with preferably low complexity and friction.

From here, we can explore other developments and tutorials on Kubernetes, such as o11y or observability (PLG, ELK, ELF, TICK, Jaeger, Pyroscope), service mesh (Linkerd, Istio, NSM, Consul Connect, Cillium), and progressive delivery (ArgoCD, FluxCD, Spinnaker).