Table of Contents
- Overview
- Project Architecture
- Step 1: Setting Up Microservices Repositories
- Step 2: Implementing CI/CD with Jenkins
- Step 3: Containerizing Microservices with Docker
- Step 4: Deploying to Kubernetes
- Step 5: Implementing Monitoring with Prometheus and Grafana
- Step 6: Implementing Logging with ELK Stack
- Conclusion
- Final Thoughts
Overview
This project will guide you through setting up an end-to-end CI/CD pipeline for a microservices-based application. You will learn how to build, deploy, monitor, and log a microservices architecture using Jenkins, Docker, Kubernetes, Prometheus, Grafana, and the ELK Stack.
Project Architecture
The project consists of the following components:
- Microservices: User Service, Order Service, Payment Service.
- Jenkins: For continuous integration and deployment.
- Docker: For containerization of services.
- Kubernetes: For orchestrating the deployment.
- Prometheus & Grafana: For monitoring and observability.
- ELK Stack: For logging and log management.
Step 1: Setting Up Microservices Repositories
1.1. Create Repositories
You need to create three separate Git repositories for each microservice:
- user-service
- order-service
- payment-service
1.2. Microservices Structure
Each repository should have the following directory structure:
- src/
- main/
- java/
- resources/
- Dockerfile
- Jenkinsfile
- pom.xml (for Maven projects) or package.json (for Node.js projects)
This structure ensures consistency across services and facilitates the CI/CD process.
1.3. Cloning Repositories
To get started, clone each repository to your local machine:
git clone https://github.com/SelimHorri/ecommerce-microservice-backend-app/user-service
git clone https://github.com/SelimHorri/ecommerce-microservice-backend-app/order-service
git clone https://github.com/SelimHorri/ecommerce-microservice-backend-app/payment-service
Step 2: Implementing CI/CD with Jenkins
2.1. Jenkins Setup
Jenkins is a key component in our CI/CD pipeline. Follow these steps to set it up:
- Install Jenkins:
wget -q -O - https://pkg.jenkins.io/debian/jenkins.io.key | sudo apt-key add -
sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt-get update
sudo apt-get install jenkins
- Start Jenkins:
sudo systemctl start jenkins
Access Jenkins:
Openhttp://localhost:8080
in your browser to access the Jenkins dashboard.-
Install Required Plugins:
- Navigate to Manage Jenkins > Manage Plugins.
- Install the following plugins:
- Docker Pipeline Plugin
- Kubernetes Plugin
- Git Plugin
2.2. Creating Jenkinsfile for CI/CD
Each microservice should have a Jenkinsfile
in its repository to define the CI/CD pipeline.
Example Jenkinsfile:
pipeline {
agent any
stages {
stage('Checkout') {
steps {
git 'https://github.com/yourusername/user-service.git'
}
}
stage('Build') {
steps {
sh 'mvn clean package'
}
}
stage('Docker Build & Push') {
steps {
script {
dockerImage = docker.build("yourdockerhub/user-service:${env.BUILD_ID}")
docker.withRegistry('https://index.docker.io/v1/', 'dockerhub-credentials') {
dockerImage.push("${env.BUILD_ID}")
dockerImage.push("latest")
}
}
}
}
stage('Deploy to Kubernetes') {
steps {
sh 'kubectl apply -f kubernetes/deployment.yaml'
}
}
}
}
This Jenkinsfile
automates the build, Docker image creation, and deployment to Kubernetes.
Step 3: Containerizing Microservices with Docker
3.1. Creating Dockerfile for Each Service
Each microservice requires a Dockerfile
to define how the container image is built.
Example Dockerfile (for a Java-based service):
# Use an official Java runtime as a parent image
FROM openjdk:11-jre-slim
# Set the working directory in the container
WORKDIR /app
# Copy the jar file into the container
COPY target/user-service.jar /app/user-service.jar
# Run the jar file
ENTRYPOINT ["java", "-jar", "/app/user-service.jar"]
# Expose the port the application runs on
EXPOSE 8080
3.2. Building and Pushing Docker Images
After creating the Dockerfile, build and push the Docker images to Docker Hub.
docker build -t yourdockerhub/user-service:latest .
docker push yourdockerhub/user-service:latest
Repeat the process for the order-service
and payment-service
.
Step 4: Deploying to Kubernetes
4.1. Setting Up Kubernetes Cluster
You can choose to deploy the application on a local Kubernetes cluster using Minikube or on AWS EKS for a production environment.
Minikube Setup:
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube
minikube start
EKS Setup (requires AWS CLI):
aws eks --region us-west-2 create-cluster --name DevOpsCluster --role-arn arn:aws:iam::YOUR_ACCOUNT_ID:role/EKSRole --resources-vpc-config subnetIds=subnet-0123456789abcdef0,subnet-abcdef0123456789
4.2. Creating Kubernetes Manifests
Create deployment and service manifests for each microservice.
Example Deployment Manifest (deployment.yaml
):
apiVersion: apps/v1
kind: Deployment
metadata:
name: user-service
spec:
replicas: 3
selector:
matchLabels:
app: user-service
template:
metadata:
labels:
app: user-service
spec:
containers:
- name: user-service
image: yourdockerhub/user-service:latest
ports:
- containerPort: 8080
Example Service Manifest (service.yaml
):
apiVersion: v1
kind: Service
metadata:
name: user-service
spec:
type: NodePort
selector:
app: user-service
ports:
- port: 8080
targetPort: 8080
nodePort: 30001
4.3. Applying Manifests
Deploy the microservices to the Kubernetes cluster using the following commands:
kubectl apply -f kubernetes/deployment.yaml
kubectl apply -f kubernetes/service.yaml
Repeat the process for the order-service
and payment-service
.
Step 5: Implementing Monitoring with Prometheus and Grafana
5.1. Installing Prometheus and Grafana
To install Prometheus and Grafana, use Helm, a package manager for Kubernetes.
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm install prometheus prometheus-community/prometheus
helm install grafana grafana/grafana
5.2. Configuring Prometheus to Scrape Kubernetes Metrics
Configure Prometheus to scrape metrics from the Kubernetes cluster by editing the prometheus.yml
file:
scrape_configs:
- job_name: 'kubernetes'
kubernetes_sd_configs:
- role: node
5.3. Accessing Grafana Dashboard
Access Grafana at http://localhost:3000
and log in with the default credentials admin/admin
. Import Prometheus as a data source and create dashboards to visualize metrics.
Step 6: Implementing Logging with ELK Stack
6.1. Deploying ELK Stack on Kubernetes
Deploy the ELK (Elasticsearch, Logstash, Kibana) stack using Helm:
helm repo add elastic https://helm.elastic.co
helm install elasticsearch elastic/elasticsearch
helm install kibana elastic/kibana
helm install logstash elastic/logstash
6.2. Configuring Logstash to Collect Kubernetes Logs
Create a Logstash configuration file to collect logs from Kubernetes:
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => ["http://elasticsearch:9200"]
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
}
}
6.3. Accessing Kibana Dashboard
Access Kibana at http://localhost:5601
and visualize logs collected from Kubernetes.
Conclusion
In this project, you learned how to set up an end-to-end CI/CD pipeline for a microservices-based application, deploy it to Kubernetes, and monitor it using Prometheus, Grafana, and ELK Stack. This setup is highly scalable and provides a robust foundation for any microservices architecture.
Final Thoughts
This project is designed to provide a comprehensive learning experience for DevOps engineers. By following this guide, you'll gain hands-on experience with key DevOps tools and practices. For more detailed content and discussions, visit the full article on Dev.to.
๐ค Author
Join Our Telegram Community || Follow me on GitHub for more DevOps content!
Top comments (5)
Thank for sharing
@mohamed_karim_2dddebb42bd ๐๐
Thanks for writeup
Thanks ๐ @kishan_vp_66a386dcc65935
1.3. Cloning Repositories
the link to the Repositories are dead.