Setting up a production Kubernetes service on AWS EKS

The easiest way to create an EKS on AWS is to use eksctl. And it is recommended to create a bastion server in AWS and run it there rather than a laptop as the environment to run the eksctl cli.

Create AWS User

First, create an admin user so that it can be used programatically in AWS IAM, and obtain the AWS Access Key ID and AWS Secret Access Key of the user.

Create Bastion Server

Let's create a bastion server in AWS. Even if the instance type is t3.small, it is sufficient.

Install kubectl

Install kubectl of desired kubernetes version

$ curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.21.2/2021-07-05/bin/linux/amd64/kubectl
$ chmod +x ./kubectl
$ mkdir -p $HOME/bin && cp ./kubectl $HOME/bin/kubectl && export PATH=$PATH:$HOME/bin
$ echo 'export PATH=$PATH:$HOME/bin' >> ~/.bashrc
$ kubectl version --short --client

Reference link: https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html

Install aws cli

To use eksctl, We need to set the credential of the AWS user.

$ apt install unzip
$ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
$ unzip awscliv2.zip
$ ./aws/install

Reference link to install the aws cli:
https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

AWS cli configuration settings

Now set configuration in aws cli in the following way-

$ aws configure
AWS Access Key ID [None]: ~~~
AWS Secret Access Key [None]: ~~~
Default region name [None]: ap-southeast-1
Default output format [None]: json

Reference link: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html

Install eksctl cli

$ curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
$ mv /tmp/eksctl /usr/local/bin
$ eksctl version
0.98.0

Reference link: https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html

Create EKS Cluster

Create EKS Cluster with eksctl cli.

$ eksctl create cluster \
--version 1.21 \
--name eks-monirul-cluster \
--vpc-nat-mode HighlyAvailable \
--node-private-networking \
--region ap-southeast-1 \
--node-type t3.medium \
--nodes 2 \
--with-oidc \
--ssh-access \
--ssh-public-key monirul \
--managed

Here,
version: the Kubernetes version to use

vpc-nat-mode: All outbounds of kubernetes go out through the nat gateway. The default option is single, so only one is created. In a development environment, it may not matter, but in production, we must use the HighAvailable option to create one for each subnet.

node-private-networking: If this option is not present, a node group is created in the public subnet. Use this option so that it is created in a private subnet for security.

node-type: the instance type of the node to be created

nodes: the number of nodes to be created

Check if the cluster has been successfully created

$ kubectl get nodes -o wide

EKS security related settings

Up to this point, a Kubernets cluster is created in a secure way considering HA for commercial use. But for security, we need to do one more thing. The current state is that the cluster endpoint is allowed to be public.
https://aws.amazon.com/blogs/containers/de-mystifying-cluster-networking-for-amazon-eks-worker-nodes/
https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html

To restrict this, allow private access and modify kubernetes to issue commands only to the bastion server by limiting the CIDR block. Enter the public IPv4 address of the bastion server.

$ eksctl utils update-cluster-endpoints --cluster=eks-monirul-cluster --private-access=true --public-access=true --approve
$ eksctl utils set-public-access-cidrs --cluster=eks-monirul-cluster 1.1.1.1/32 --approve

1.1.1.1/32 is the address of the bastion server.

In fact, EKS Cluster creation can be created at once by creating a yaml file with the --dry-run option of eksctl and giving all options from creation to EKS security-related settings.

Now, if we install Istio in the EKS Cluster created in this way and deploy the service, we can use it immediately.

Argo CD installation, setup & deploy a simple app with kustomize

ArgoCD monitors changes in Kubernetes manifests managed by GitOps, and plays a role in maintaining the form deployed in the actual cluster in the same way.

Argo CD installation

https://argo-cd.readthedocs.io/en/stable/getting_started/
Install Argo CD on kubernetes cluster

In production, install the HA version of the Argo CD.

$ kubectl create namespace argocd
$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/ha/install.yaml

Install Argo CD CLI

Install Argo CD CLI

$ curl -sSL -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
$ chmod +x /usr/local/bin/argocd

Argo CD service exposure

Argo CD does not expose the server to the outside by default. Change the service type to LoadBalancer as shown below and expose it to the outside.

$ kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

Change admin password

Argo CD stores the initial password of the initial admin account as the secret of kubernetes. Get the password as below.

$ kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
jaIyQ3MMuLnl6h0l

Log in to Argo CD using the Argo CD CLI. First, get the address of the created Load Balancer.

$ kubectl get svc argocd-server -n argocd

And log in. Username is admin

$ argocd login <ARGOCD_SERVER_DOMAIN/URL>

Update the password of the admin user after first login.

$ argocd account update-password

Argo CD polls the Git repository once every 3 minutes to check the difference from the actual kubernetes cluster. Therefore, if we are unlucky during distribution, we have to wait up to 3 minutes before Argo CD distributes the changed image. If we want to eliminate the delay caused by polling like this, we can create a webhook with Argo CD in the Git repository. Here is the link:
https://argo-cd.readthedocs.io/en/stable/operator-manual/webhook/

And, usually Argo CD has only a specific IP inbound in the security group to prevent access except for internal developers or operators. At this time, if we have created a webhook as above, we must also open Github's webhook-related API inbound to the Argo CD's load balancer.

The link below contains information about the IP address of GitHub,
https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/about-githubs-ip-addresses
We can check the IP that needs to be put in the actual inbound below. Just put the list in hooks.
https://api.github.com/meta

If we configure to send notification to Slack using Argo CD Notification, the development team can receive a notification when the deployment is not successful.

There is also a way to set up the project by using the ApplicationSet of the Argo CD. If we used App of Apps in the past, a more advanced concept here is Application Sets.

https://opengitops.dev/
https://github.com/open-gitops/documents

Currently, kustomize is used for configuration management of Kubernetes, and kustomize is deployed from Argo CD. If we are using branching in kustomize or helm, it might be helpful to read Stop Using Branches for Deploying to Different GitOps Environments.

Kustomize yaml-

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namePrefix: kustomize-monirul-

resources:
- nginx-deployment.yaml
- nginx-svc.yaml

Deployment definition file: nginx-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx

Service definition file: nginx-svc.yaml

apiVersion: v1

kind: Service

metadata:

  labels:

    app: nginx

  name: nginx

spec:

  ports:


port: 80
protocol: TCP
targetPort: 80
selector:
app: nginx
type: ClusterIP

ArgoCD configuration to deploy simple app:

Screenshot of Argo CD: