Unlocking the Secrets of AWS EKS: How I Built a Scalable and Resilient User and Email Service Application

Introduction:

In today's fast-paced digital world, building a scalable and resilient application is crucial for any business. As more and more companies are moving their workloads to the cloud, it's essential to have a solid understanding of cloud technologies and best practices. In this blog post, I will show you how I built a scalable and resilient User and Email Service application on AWS EKS.

Background:

Recently, I worked on a project to create a User and Email Service application that allows users to sign up for a new account and receive a welcome email. The application was built using Python, Kafka Streams, and Kubernetes. The User Service publishes a message on a "Provision User" topic, and the Email Service consumes this message about a new user and sends a welcome email to them. The User and Email Services did not have to directly message each other, but their respective jobs were executed asynchronously.

Building on AWS EKS:

AWS EKS (Elastic Kubernetes Service) is a managed Kubernetes service that makes it easy to deploy, scale, and manage containerized applications using Kubernetes on AWS. To build our User and Email Service application on AWS EKS, we first had to create an EKS cluster. We used the AWS management console to create a new EKS cluster named "my-eks-cluster" and selected the desired VPC and subnets for the cluster. We also used an existing security group named "eks-cluster-sg" and an existing IAM role named "eks-cluster-role" for the cluster.

Next, we had to deploy a Kafka cluster on the EKS cluster. We used the Helm chart to deploy the Kafka cluster on the EKS cluster and created a namespace named "kafka-ns" for the kafka cluster. We also created a values.yaml file for the chart, specifying the number of replicas, resources, and other configuration options. We used Helm to install the chart, passing in the values file and the namespace "kafka-ns".

After deploying the Kafka cluster, we created and configured the necessary Kubernetes resources, such as Services and Deployments, to deploy the User service and Email service to the EKS cluster. We created a new namespace named "user-email-ns" for the User service and Email service. We created a Deployment resource for each service, specifying the number of replicas, resources, and other configuration options. We also created a Service resource for each service, specifying the desired type (ClusterIP, NodePort, LoadBalancer, etc.) and other configuration options. We then applied the resources to the cluster using kubectl or Helm.

We also created and configured an AWS Elastic Load Balancer to route traffic to the User service and Email service. We used an existing Elastic Load Balancer named "my-elb" and updated the Service resource for each service to use the load balancer "my-elb".

For persistence and other dependencies, we used an existing RDS instance named "my-rds" and an existing S3 bucket named "my-s3-bucket". We also created and configured the necessary IAM roles and policies to allow the User service and Email service to access the necessary AWS resources.

For monitoring, we set up alarms and autoscaling rules, and also created and configured a CI/CD pipeline to automate the deployment of new versions of the application.

Here is an example of a full production environment code package that uses Python, Kafka Streams, and Kubernetes to create a new user in an application:

Code:

1. User Service:

from kafka import KafkaProducer

producer = KafkaProducer(bootstrap_servers='kafka-cluster:9092')

def create_user(user_data):
    producer.send('Provision User', key=user_data['id'], value=user_data)

1. Email Service:

from kafka import KafkaConsumer
from email.mime.text import MIMEText
import smtplib

consumer = KafkaConsumer('Provision User', bootstrap_servers='kafka-cluster:9092')

def send_welcome_email(user_data):
    message = MIMEText('Welcome, {}!'.format(user_data['name']))
    message['To'] = user_data['email']
    message['Subject'] = 'Welcome to our service!'
    smtp_server = smtplib.SMTP('smtp.example.com')
    smtp_server.send_message(message)

for msg in consumer:
    send_welcome_email(msg.value)

Kubernetes Configuration:

apiVersion: v1
kind: Service
metadata:
  name: kafka-cluster
  labels:
    app: kafka-cluster
spec:
  ports:
    - name: kafka
      port: 9092
      targetPort: 9092
  selector:
    app: kafka-cluster

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-cluster
spec:
  replicas: 3
  selector:
    matchLabels:
      app: kafka-cluster
  template:
    metadata:
      labels:
        app: kafka-cluster
    spec:
      containers:
        - name: kafka
          image: confluentinc/cp-kafka:5.5.1
          ports:
            - containerPort: 9092
          env:
            - name: KAFKA_ADVERTISED_LISTENERS
              value: PLAINTEXT://kafka-cluster:9092
            - name: KAFKA_ZOOKEEPER_CONNECT
              value: zookeeper-cluster:2181

apiVersion: v1
kind: Service
metadata:
  name: user-service
  labels:
    app: user-service
spec:
  ports:
    - name: http
      port: 80
      targetPort: 80
  selector:
    app: user-service

apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: user-service
  template:
    metadata:
      labels:
        app: user-service
    spec:
      containers:
        - name: user-service
          image: python:3.9
          ports:
            - containerPort: 80

apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-service
  namespace: user-email-ns
spec:
  replicas: 3
  selector:
    matchLabels:
      app: user-service
  template:
    metadata:
      labels:
        app: user-service
    spec:
      containers:
      - name: user-service
        image: python:3.9
        ports:
        - containerPort: 80

This configuration creates a Deployment named "user-service" in the "user-email-ns" namespace. The Deployment creates 3 replicas of a Pod running the "python:3.9" image and exposing port 80.

Here is an example of the Kubernetes configuration for the User service Service:

apiVersion: v1
kind: Service
metadata:
  name: user-service
  namespace: user-email-ns
spec:
  selector:
    app: user-service
  ports:
  - name: http
    port: 80
    targetPort: 80
  type: LoadBalancer

This configuration creates a Service named "user-service" in the "user-email-ns" namespace. The Service routes traffic to Pods with the label "app: user-service" on port 80 and exposes the service on a LoadBalancer.

In this blog post, I have shown you how I built a scalable and resilient User and Email Service application on AWS EKS. We have used Python, Kafka Streams, and Kubernetes to build the application and AWS EKS to deploy it to the cloud. We also used various other AWS services such as RDS, S3, and Elastic Load Balancer to provide persistence and other dependencies. This is just a small example of what is possible with AWS EKS and the possibilities are endless. It's important to continuously learn new cloud technologies and best practices to stay ahead in the game.

NOTE: This project is overly simplified and does not accurately reflect the complexity and challenges of building a production-ready application on AWS EKS.