In this guide, we'll explore how to create a DevOps monitoring and automation system using Jenkins for CI/CD, Prometheus for metrics collection, Grafana for data visualization, and Docker for containerization.
Installing Docker
First, ensure Docker is installed on your system. Follow the installation instructions provided on the Docker website for your specific operating system.
Setting Up Jenkins
To run Jenkins in a Docker container, execute the following command:
docker run -u 0 --privileged --name jenkins -it -d -p 8080:8080 -p 50000:50000 \
-v /var/run/docker.sock:/var/run/docker.sock \
-v $(which docker):/usr/bin/docker \
-v /home/jenkins_home:/var/jenkins_home \
jenkins/jenkins:latest
Access and Configure Jenkins
Open your browser and navigate to http://localhost:8080.
Provide the initial admin password, which can be found by running docker logs .
Install the suggested plugins during the initial setup.
From the Jenkins dashboard, create a new item and choose "Pipeline".
Install Required Plugins
Navigate to "Manage Jenkins" > "Manage Plugins" and install the following plugins:
Docker Plugin
Docker Pipeline Plugin
Prometheus Metrics Plugin
Restart Jenkins if prompted.
Setting Up Prometheus and Grafana
Create a Docker Network
docker network create monitoring
Run Prometheus
Create a prometheus.yml configuration file with the following content:
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'jenkins'
static_configs:
- targets: ['jenkins:8080']
metrics_path: '/prometheus'
- job_name: 'node'
static_configs:
- targets: ['node-exporter:9100']
Run Prometheus with this configuration:
docker run -d --name=prometheus --network=monitoring -p 9090:9090 \
-v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus
Run Node Exporter
docker run -d --name=node-exporter --network=monitoring --restart=unless-stopped prom/node-exporter
Run Grafana
docker run -d --name=grafana --network=monitoring -p 3000:3000 grafana/grafana
Configuring Grafana
Access Grafana
Open your browser and go to http://localhost:3000. Log in with the default credentials (username: admin, password: admin).
Add Prometheus as a Data Source
Navigate to "Data Sources" on the left sidebar, click "Add data source," and select "Prometheus".
Enter http://prometheus:9090 as the Prometheus server URL.
Click "Save & Test".
Create a Dashboard
Select "Dashboards" from the left sidebar to create a new dashboard.
Import a dashboard using an ID or URL (e.g., ID: 11159).
Add a query to visualize Jenkins job duration:
default_jenkins_builds_last_build_duration_milliseconds
Automating Application Deployment with Jenkins
Set Up a Jenkins Pipeline
From your Jenkins dashboard, navigate to your pipeline project, go to "Configure," and add the following script under "Pipeline Script":
pipeline {
agent any
environment {
DB_HOST = 'your-db-host'
DB_USER = 'your-db-user'
DB_PASS = 'your-db-password'
DB_NAME = 'your-db-name'
}
stages {
stage('Checkout') {
steps {
script {
checkout([$class: 'GitSCM', branches: [[name: '*/main']],
doGenerateSubmoduleConfigurations: false,
extensions: [[$class: 'CleanCheckout']],
submoduleCfg: [],
userRemoteConfigs: [[url: 'https://github.com/yourusername/your-nodejs-typescript-api-repo.git',
credentialsId: 'git-credentials-id']]
])
}
}
}
stage('Install Dependencies') {
steps {
sh 'npm install'
}
}
stage('Build') {
steps {
sh 'npm run build'
}
}
stage('Test') {
steps {
sh 'npm test'
}
}
stage('Build Docker Image') {
steps {
script {
sh 'docker build -t your-dockerhub-username/your-nodejs-typescript-api:latest .'
}
}
}
stage('Push Docker Image') {
steps {
script {
withCredentials([usernamePassword(credentialsId: 'dockerhub-credentials', usernameVariable: 'DOCKER_USERNAME', passwordVariable: 'DOCKER_PASSWORD')]) {
sh 'echo $DOCKER_PASSWORD | docker login -u $DOCKER_USERNAME --password-stdin'
}
sh 'docker push your-dockerhub-username/your-nodejs-typescript-api:latest'
}
}
}
stage('Deploy') {
steps {
script {
sh '''
docker run -d -p 3000:3000 \
-e NODE_ENV=$NODE_ENV \
-e DB_HOST=$DB_HOST \
-e DB_USER=$DB_USER \
-e DB_PASS=$DB_PASS \
-e DB_NAME=$DB_NAME \
your-dockerhub-username/your-nodejs-typescript-api:latest
'''
}
}
}
stage('Monitor') {
steps {
script {
// Additional monitoring setup can be added here
}
}
}
}
post {
success {
echo 'Build succeeded!'
}
failure {
echo 'Build failed!'
}
}
}
Adding Credentials
Add credentials for GitHub and Docker in Jenkins:
Navigate to "Manage Jenkins" > "Credentials".
Select "Global" and add your credentials for GitHub and Docker. Note the ID, as it will be referenced in the pipeline script.
By following these steps, you can set up a comprehensive DevOps monitoring and automation tool using Jenkins, Prometheus, Grafana, and Docker.
Build Application
Build your application and view the metrics on the Grafana dashboard 😃.
Top comments (5)
Hmm, Grafana does not seem to see prometheus. I get:
Docker looks like:
In addition, I get:
I wonder, where's a good place for the prometheus.yml file? I just put it in ~ as I am just testing things out.
I'm new to this but
metrics_pathline moved above itsstatic_configsline and a de-dent as it cannot be parsed.~/prometheus/prometheus.ymland run:The container now stays running, ...on with the rest of it.
Hello ✋
I have one query regarding to automated testing:
"What are the most experience challenges faced by teams while using automated testing in ci/cd pipeline?"