This may come as a surprise to some, but the AWS managed Kubernetes service, EKS (Elastic Kubernetes Service), no longer comes with support for storage through Kubernetes persistent volumes, which is required for many applications, such as databases.
You will need to add a (CSI) driver to enable persistent volumes support.
This guide, Part 2 of Ultimate EKS Baseline Cluster series, covers doing just that by installing CSI) called AWS EBS CSI Driver.
0. Prerequisites
These are are some prerequisites and initial steps needed to get started before provisioning a Kubernetes cluster and installing add-ons.
0.1 Knowledge: Systems
Basic concepts of systems, such as Linux and the shell (redirection, pipes, process substitution, command substitution, environment variables), as well as virtualization and containers are useful. The concept of a service (daemon) is important.
0.2 Knowledge: Kubernetes
In Kubernetes, familiarity with service types: ClusterIP, NodePort, LoadBalancer, ExternalName and the ingress resource are important.
Exposure to other types of Kubernetes resource objects used in this guide are helpful: persistentvolumeclaims (pvc), storageclass (sc), pods, deployments, statefulsets (sts), configmaps, serviceaccount (sa) and networkpolicies.
0.3 Tools
These are the tools used in this article series:
-
AWS CLI [
aws] is a tool that interacts with AWS. -
kubectl client [
kubectl] a the tool that can interact with the Kubernetes cluster. This can be installed using adsf tool. - eksctl [eksctl] is the tool that can provision EKS cluster as well as supporting VPC network infrastructure.
-
POSIX Shell [
sh] such as bash[bash] or zsh [zsh] are used to run the commands.
These tools are highly recommended:
-
adsf [
adsf] is a tool that installs versions of popular tools like kubectl. -
jq [
jq] is a tool to query and print JSON data -
GNU Grep [
grep] supports extracting string patterns using extended Regex and PCRE.
0.3 Setup Environment Variables
These environment variables will be used throughout this guide. If opening up a new browser tab, make sure to set the environment variables accordingly.
# variables used to create EKS
export AWS_PROFILE="my-aws-profile" # CHANGEME
export EKS_CLUSTER_NAME="my-unique-cluster-name" # CHANGEME
export EKS_REGION="us-west-2"
export EKS_VERSION="1.26"
# KUBECONFIG variable
export KUBECONFIG=$HOME/.kube/$EKS_REGION.$EKS_CLUSTER_NAME.yaml
# account id
export ACCOUNT_ID=$(aws sts get-caller-identity \
--query "Account" \
--output text
)
# ebs-csi-driver
export ROLE_NAME_ECSI="${EKS_CLUSTER_NAME}_EBS_CSI_DriverRole"
export ACCOUNT_ROLE_ARN_ECSI="arn:aws:iam::$ACCOUNT_ID:role/$ROLE_NAME_ECSI"
POLICY_NAME_ECSI="AmazonEBSCSIDriverPolicy" # preinstalled by AWS
export POLICY_ARN_ECSI="arn:aws:iam::aws:policy/service-role/$POLICY_NAME_ECSI"
0.4 AWS Setup
There's an assumption that AWS CLI have been setup and configured with a profile. This is required before creating or interacting with an EKS cluster.
You can test access to a configured profile with the following command:
export AWS_PROFILE="<your-profile-goes-here>"
aws sts get-caller-identity
This should show something like the following with values appropriate to your environment, e.g. example output to IAM user named kwisatzhaderach:
{
"UserId": "AIDAXXXXXXXXXXXXXXXXX",
"Account": "XXXXXXXXXXXX",
"Arn": "arn:aws:iam::XXXXXXXXXXXX:user/kwisatzhaderach"
}
0.5 EKS Cluster
This article requires that an EKS cluster has been previously provisioned using eksctl tool.
I wrote a previous article that covered this how to provision EKS with two commands:
0.5.1 Existing EKS Cluster
If you have an existing EKS cluster, but need to configure KUBECONFIG for access, you can run this:
mkdir -p $HOME/.kube # conditionally create ~/.kube
# use consistent $KUBECONFIG
export KUBECONFIG=$HOME/.kube/$EKS_REGION.$EKS_CLUSTER_NAME.yaml
# update config pointed to by $KUBECONFIG
aws eks update-kubeconfig \
--name $CLUSTER \
--region $REGION \
--profile $PROFILE
0.5.2 Create a new EKS Cluster
If you have not setup an EKS cluster, you can set it up with the following commands (~20 minutes process):
mkdir -p $HOME/.kube # conditionally create ~/.kube
# use consistent $KUBECONFIG
export KUBECONFIG=$HOME/.kube/$EKS_REGION.$EKS_CLUSTER_NAME.yaml
# provision EKS + add config to $KUBECONFIG
eksctl create cluster \
--version $EKS_VERSION \
--region $EKS_REGION \
--name $EKS_CLUSTER_NAME \
--nodes 3
# setup OIDC provider for least privilege
eksctl utils associate-iam-oidc-provider \
--cluster $EKS_CLUSTER_NAME \
--region $EKS_REGION \
--approve
Details of this were covered in the previous article
0.6 Kubernetes Client Setup
If you use asdf to install kubectl, you can get the latest version with the following:
# install kubectl plugin for asdf
asdf plugin-add kubectl \
https://github.com/asdf-community/asdf-kubectl.git
asdf install kubectl latest
# fetch latest kubectl
asdf install kubectl latest
asdf global kubectl latest
# test results of latest kubectl
kubectl version --short --client 2> /dev/null
This should show something like:
Client Version: v1.27.4
Kustomize Version: v5.0.1
1. AWS EBS CSI driver
The current versions of EKS starting with 1.23 no longer come with a persistent volume support, so you have to install it on your own. The best method or at least the easiest way to install this, is using EKS add-ons facility. This will install the EBS CSI driver.
Installation of this component will require the following steps:
- Create IAM Role (e.g.
EBS_CSI_DriverRole) and associate it to Kubernetes service account (i.e.ebs-csi-controller-sa). - Deploy AWS EBS CSI driver using EKS add-ons facility, which also sets up the Kubernetes service account (i.e.
ebs-csi-controller-sa) with an association back to the above IAM Role (e.g.EBS_CSI_DriverRole). - Create storage class that uses new the EBS CSI driver
1.1 Setup IAM Role and K8S SA association
The following process will create an IAM Role with permissions to access AWS EBS API. The service account ebs-csi-controller-sa will be created later when installing the driver.
# AWS IAM role bound to a Kubernetes service account
eksctl create iamserviceaccount \
--name "ebs-csi-controller-sa" \
--namespace "kube-system" \
--cluster $EKS_CLUSTER_NAME \
--region $EKS_REGION \
--attach-policy-arn $POLICY_ARN_ECSI \
--role-only \
--role-name $ROLE_NAME_ECSI \
--approve
This will create a IAM Role, which you can verify with:
aws iam get-role --role-name $ROLE_NAME_ECSI
This should show something like this:
{
"Role": {
"Path": "/",
"RoleName": "mycluster_EBS_CSI_DriverRole",
"RoleId": "AROAZYKZFDW7YZGW3Q5S7",
"Arn": "arn:aws:iam::XXXXXXXXXXXX:role/mycluster_EBS_CSI_DriverRole",
"CreateDate": "2023-07-07T21:19:32+00:00",
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::XXXXXXXXXXXX:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/6311CF96A267242FD6587B1C29D57F1D"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-west-2.amazonaws.com/id/6311CF96A267242FD6587B1C29D57F1D:aud": "sts.amazonaws.com",
"oidc.eks.us-west-2.amazonaws.com/id/6311CF96A267242FD6587B1C29D57F1D:sub": "system:serviceaccount:kube-system:ebs-csi-controller-sa"
}
}
}
]
},
"Description": "",
"MaxSessionDuration": 3600,
"Tags": [
{
"Key": "alpha.eksctl.io/cluster-name",
"Value": "mycluster"
},
{
"Key": "eksctl.cluster.k8s.io/v1alpha1/cluster-name",
"Value": "mycluster"
},
{
"Key": "alpha.eksctl.io/iamserviceaccount-name",
"Value": "kube-system/ebs-csi-controller-sa"
},
{
"Key": "alpha.eksctl.io/eksctl-version",
"Value": "0.141.0-dev+5c8318ed5.2023-05-12T11:33:48Z"
}
],
"RoleLastUsed": {}
}
}
1.2 Install AWS EBS CSI Drvier
This installation uses the EKS Addons feature to install the component.
# Install Addon
eksctl create addon \
--name "aws-ebs-csi-driver" \
--cluster $EKS_CLUSTER_NAME \
--region $EKS_REGION \
--service-account-role-arn $ACCOUNT_ROLE_ARN_ECSI \
--force
# Pause here until driver is active
ACTIVE=""; while [[ -z "$ACTIVE" ]]; do
if eksctl get addon \
--name "aws-ebs-csi-driver" \
--region $EKS_REGION \
--cluster $EKS_CLUSTER_NAME \
| tail -1 \
| awk '{print $3}' \
| grep -q "ACTIVE"
then
ACTIVE="1"
fi
done
It is important to wait until status changes to ACTIVE before proceeding.
You can inspect the pods created by running the following command:
kubectl get pods \
--namespace "kube-system" \
--selector "app.kubernetes.io/name=aws-ebs-csi-driver"
This should show something like:
NAME READY STATUS RESTARTS AGE
ebs-csi-controller-6d5b7bfd56-bwr5x 6/6 Running 0 2m1s
ebs-csi-controller-6d5b7bfd56-wtxf6 6/6 Running 0 2m2s
ebs-csi-node-hjmf5 3/3 Running 0 2m2s
ebs-csi-node-tzpgs 3/3 Running 0 2m2s
You can verify the service account annotations references the IAM Role for the EBS CSI driver.
kubectl get serviceaccount "ebs-csi-controller-sa" \
--namespace "kube-system" \
--output yaml
This should show something like the following
apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::XXXXXXXXXXXX:role/my-unique-cluster-name_EBS_CSI_DriverRole
creationTimestamp: "2023-07-24T21:12:02Z"
labels:
app.kubernetes.io/component: csi-driver
app.kubernetes.io/managed-by: EKS
app.kubernetes.io/name: aws-ebs-csi-driver
app.kubernetes.io/version: 1.21.0
name: ebs-csi-controller-sa
namespace: kube-system
resourceVersion: "1922"
uid: 1b8ecfdb-ce64-491c-8ced-e08b5519755c
1.2.1 Sidebar: eks-addons vs helm chart?
The CSI driver can be installed using either EKS Addons facility from the previous step, or using the aws-ebs-csi-driver Helm chart. I prefer the EKS addons because of simplicity, but also because it installs an extra snapshot container that doesn't come by default with the helm chart.
I documented the full process using the helm chart in this article:
1.3 Create storage class that uses the EBS CSI driver
In order to use the driver, we will need to create a storage class. You can do so by running the following command:
# create ebs-sc storage class
cat <<EOF | kubectl apply --filename -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ebs-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
EOF
When completed you can verify the storage class was created with:
kubectl get storageclass
This should show something like this:
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
ebs-sc ebs.csi.aws.com Delete WaitForFirstConsumer true 17s
gp2 (default) kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 30m
1.4 Set new storage class to the default (optional)
This is an optional step. As there’s no functional default storage class, we can set the newly created storage class to be the default with the following commands:
kubectl patch storageclass gp2 --patch \
'{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
kubectl patch storageclass ebs-sc --patch \
'{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
After this, you can verify the change with:
kubectl get storageclass
This should show something like:
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
ebs-sc (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 31s
gp2 kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 30m
1.5 Testing Persistent Volume
In this small test, we deploy a pod that continually writes to the external volume, and a volume claim to allocate storage using the storage class we created earlier.
If this works, the storage will be provisioned in the cloud to create the volume, and then it will be attached to the node and mounted in the pod. If this fails, you will see that the pod will be stuck in pending mode.
# create pod with persistent volume
kubectl create namespace "ebs-test"
# deploy application with mounted volume
cat <<-'EOF' | kubectl apply --namespace "ebs-test" --filename -
apiVersion: v1
kind: Pod
metadata:
name: app
spec:
containers:
- name: app
image: ubuntu
command: ["/bin/sh"]
args: ["-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 5; done"]
volumeMounts:
- name: persistent-storage
mountPath: /data
volumes:
- name: persistent-storage
persistentVolumeClaim:
claimName: ebs-claim
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ebs-claim
spec:
accessModes:
- ReadWriteOnce
storageClassName: ebs-sc
resources:
requests:
storage: 4Gi
EOF
You can test the results of the volume creation with the following command:
kubectl get all,pvc --namespace "ebs-test"
We can also look at the events that took place in this namespace with:
kubectl events --namespace "ebs-test"
This will show something like this:
LAST SEEN TYPE REASON OBJECT MESSAGE
56s Warning FailedScheduling Pod/app 0/3 nodes are available: persistentvolumeclaim "ebs-claim" not found. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod..
55s Normal WaitForPodScheduled PersistentVolumeClaim/ebs-claim waiting for pod app to be scheduled
54s Normal Provisioning PersistentVolumeClaim/ebs-claim External provisioner is provisioning volume for claim "ebs-test/ebs-claim"
54s (x2 over 54s) Normal ExternalProvisioning PersistentVolumeClaim/ebs-claim waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
50s Normal Scheduled Pod/app Successfully assigned ebs-test/app to ip-192-168-22-207.us-west-2.compute.internal
50s Normal ProvisioningSucceeded PersistentVolumeClaim/ebs-claim Successfully provisioned volume pvc-39b9cf94-8b35-436c-b56d-e2fa587245ee
48s Normal SuccessfulAttachVolume Pod/app AttachVolume.Attach succeeded for volume "pvc-39b9cf94-8b35-436c-b56d-e2fa587245ee"
44s Normal Pulling Pod/app Pulling image "ubuntu"
41s Normal Pulled Pod/app Successfully pulled image "ubuntu" in 2.952872938s (2.952907339s including waiting)
41s Normal Created Pod/app Created container app
41s Normal Started Pod/app Started container app
1.6 Delete test application
kubectl delete pod app --namespace "ebs-test"
kubectl delete pvc ebs-claim --namespace "ebs-test"
kubectl delete ns "ebs-test"
2.0 Deleting EKS
When deleting up the EKS cluster, you may want to run through these steps.
2.1 Deleting persistent volume claims
When deleting EKS cluster, if you did not delete persistent volumes, you will have left over unused EBS volumes eating costs.
You should delete all the persistent volume claims, which will delete associate persistent volumes. You can list all of the persistentvolumeclaim resources with the following command:
kubectl get persistentvolumeclaim \
--all-namespaces | grep -v none
Not that some of these will not be deleted if there is an application running that has a lock to the storage. So you will need to delete the associated application as well.
2.2 Reset Default to original Storage Class
As a precaution, we don’t want to have any resources locked that may prevent deletion of the Kubernetes cluster. Run this command if we changed the defaults earlier.
kubectl patch storageclass ebs-sc --patch \
'{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
kubectl patch storageclass gp2 --patch \
'{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
2.3 IAM Roles
These should be removed when removing Kubernetes with eksctl, but it is a good practice to remove them just in case.
eksctl delete iamserviceaccount \
--name "ebs-csi-controller-sa" \
--namespace "kube-system" \
--cluster $EKS_CLUSTER_NAME \
--region $EKS_REGION
2.4 Kubernetes cluster
Finally, the Kubernetes cluster itself can be deleted.
eksctl delete cluster \
--region $EKS_REGION \
--name $EKS_CLUSTER_NAME
Conclusion
This second article shows add support for storage, called persistent volumes on Kubernetes.
The automation using the eksctl will do the following additional things besides provisioning EKS:
- setup restricted access to AWS cloud resources using IRSA, which associates KSA with IAM Role using an OIDC provider.
- install applications using EKS addons facility, specifically the EBS CSI driver.
In future articles, I will cover how to add support for load balancers and network policies, as well as a demo application that demonstrates all of these features.
Top comments (0)