Continous Integration And Continous Deployment Of A Full-Stack Docker-Compose Application

Achieving a fully automated setup for infrastructure and application deployment is usually not complete without a proper Continuous Integration and Continuous Deployment (CI/CD) setup. Previously, I had a setup where Docker Compose was leveraged to deploy a full satck application (Python, React, MySQL) and a corresponding monitoring stack (Prometheus, Grafana, Loki and CAdvisor) for monitoring the server and container metrics and logs. Terraform (IAC) and Ansible (Configuration management) was used together to quickly spin up the services.

But to keep track of future updates to the application or state of infrastructure running the application, it’s crucial to leverage a version control system like GitHub. Also, changes to our code and configuration should be seamlessly integrated into the currently version. This is where using Github actions comes in.

GitHub Actions is a CI/CD platform provided by GitHub that enables you to automate, build, test, and deploy your code directly from your GitHub repository. It allows developers to define custom workflows using YAML files that respond to specific events in a repository, such as pushes, pull requests, or issues.

Setting up a deployment pipeline does not require a particularly steep learning curve, but as with many complicated things, requires good planning and proper design. This would ensure the many moving parts work together. To learn more about github actions, you can use this link.

Project Requirements

For this project, we will be setting up a CI/CD pipelines to automate infrastructure and application deployments and it will include:

• Cloud cost optimization with cost estimation tools like InfraCost.
• Terraform and Ansible integration for infrastructure management and monitoring stack setup.
• Git branching strategies for streamlined CI/CD pipelines.

To be able to follow along the implementation, you should have:

• Knowledge of Docker, Git, Terraform and Ansible
• An Azure subscription

The application code, configuration and workflow files for this project can be found in this github repository.

Architectural Overview

Architecture Diagram of the Setup

Implementation Process

1. Clone the repository
The repository contains code and configurations setup in directories as follows:

• .github/worflows - workflow files
• terraform - configurations for provisioning infrastructure resources on Azure
• ansible - ansible playbooks and roles for configuring the server and bringing up monitoring and application stack
• docker-compose - docker compose configuration files for both application and monitoring stack
• app - contains application code for frontend and backend with their corresponding dockerfiles

2. Workflow Configurations & branch setup
There are 6 github actions workflow files:

• terraform-validate.yml
• terraform-plan.yml
• terraform-apply.yml
• ansible-monitoring.yml
• ci-application.yml
• cd-application.yml

Create 4 branches:

• infra_features
• infra_main
• integration
• deployment

3. Infrastructure Workflow

• Push to infra_features branch -- the validate branch runs to confirm validity of terraform code

• On pull request to infra_main branch, the plan workflow runs and adds an infracost comment containing the estimated monthly cost of the new infrastructure updates

• On merge to infra_main branch, the apply workflow runs, creating or updating resources on Azure (or cloud service provider of choice).

• On successful completion of terraform apply, the ansible workflow runs. This workflow will run ansible to configure the created server infrastructure and deploy the monitoring stack on the server.
  

4. Deployment Workflow

• For deploying the application stack, when changes are made to the application code and there is a push of the new code to the integration branch, a workflow is triggered. It:
• builds new docker images
• tags and pushes the images to dockerhub

• updates docker compose files with the new tags

• On successful merge from integration branch to deployment branch, ansible is once again used to deploy the new version of the update docker compose files on the server.

Challenges and Solutions

• Managing and Authenticating with Multiple Tools, Platforms and Frameworks

• Testing Pipeline updates

Results and Learnings

• Streamlined Infrastructure Management: The infrastructure pipeline automated the provisioning of cloud resources and deployed a monitoring stack efficiently.

• Cost Transparency: Using InfraCost provided visibility into cloud expenses, enabling better decision-making.

• Improved CI/CD Efficiency: The application pipeline successfully automated Docker image builds, updates, and deployments, significantly reducing manual efforts and errors.

• End-to-End Automation: Combined workflows for infrastructure and application deployments achieved a fully automated system with minimal downtime during updates.

Conclusion

This project demonstrated the power of CI/CD pipelines in automating both infrastructure and application deployments. By integrating multiple tools into a cohesive workflow, the team achieved faster deployments, better cost control, and a more reliable system overall. The challenges provided valuable lessons on tool integration, branching strategies, and automation best practices, making future projects more efficient and scalable.