Afeez Oluwashina Adeboye

Posted on Dec 15

Implementing a Complete GitOps Pipeline: Infrastructure Provisioning and Application Deployment Automation

#terraform #ansible #githubactions #aws

Introduction

In this article, I'll walk you through implementing a complete GitOps pipeline that handles both infrastructure provisioning and application deployment. We'll use a combination of Terraform, Ansible, and Docker to create a fully automated CI/CD pipeline that follows best practices and maintains clear separation of concerns.

🔗 Project Repository: DevOps Pipeline Project

You can find all the code and configurations discussed in this article in the repository above. Feel free to star ⭐ it if you find it helpful!

In modern DevOps practices, automation is key to maintaining reliable and efficient software delivery. This project demonstrates the implementation of a comprehensive GitOps pipeline that handles both infrastructure provisioning and application deployment through automated workflows. By leveraging tools such as Terraform, Ansible, and Docker, along with GitHub Actions, we've created a system that ensures consistent infrastructure deployment, cost-aware planning, and automated application updates.

The solution addresses common challenges in DevOps practices:

How to maintain infrastructure as code with proper validation and cost control
How to automate monitoring system deployment alongside infrastructure
How to manage application deployments with proper versioning and rollout
How to keep infrastructure and application deployments separate yet coordinated

Architecture Overview

Project Overview

Core Components and Workflow Structure

The project is organized into two main pipelines:

Infrastructure Pipeline (infra_features → infra_main)
- Handles all infrastructure provisioning through Terraform
- Automatically deploys monitoring stack using Ansible
- Includes cost estimation for infrastructure changes
- Located in main branch for centralized workflow management and also in each branch for proper triggers
Application Pipeline (integration → deployment)
- Manages application container builds and deployments
- Handles Docker image versioning and updates
- Automates deployment to provisioned infrastructure

Workflow Triggers and Placement

Infrastructure Workflows:
- terraform-validate.yml: Triggers on push to infra_features
- terraform-plan.yml: Triggers on PR to infra_main
- terraform-apply.yml: Triggers on PR merge to infra_main
- ansible-monitoring.yml: Triggers after successful terraform apply
Application Workflows:
- ci-application.yml: Triggers on push to integration
- cd-application.yml: Triggers on PR merge to deployment

Expected Outcomes

Infrastructure Pipeline:
- Automated validation of Terraform configurations
- Cost estimation in PR comments
- Provisioned AWS infrastructure
- Deployed monitoring stack (Prometheus, Grafana, etc.)
Application Pipeline:
- Built and versioned Docker images
- Updated docker-compose configurations
- Deployed application stack to infrastructure

The separation of concerns is maintained through:

Different branches for infrastructure and application code
Separate workflows for different stages of deployment
Clear triggers that prevent unintended deployments

This structure ensures that:

Infrastructure changes are validated and cost-estimated before deployment
Application deployments only occur on properly provisioned infrastructure
Monitoring is always in place before application deployment
Changes can be tracked and reversed if needed.

Infrastructure Configuration Pipeline Deep Dive

terraform-validate.yml

This workflow is our first quality gate for infrastructure changes. While it exists in the main branch for centralization, it must also exist in the infra_features branch to properly trigger on push events.

name: Terraform Validate
run-name: ${{ github.actor }} triggered Terraform validation

on:
  workflow_dispatch: # Allows manual trigger
  push:
    branches:
      - infra_features # Trigger on pushes to infra_features branch
    paths:
      - 'terraform/**' # Only trigger if files in the terraform directory change

env:
  # AWS Credentials
  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
  TF_VAR_aws_region: ${{ vars.AWS_REGION }}

jobs:
  validate:
    name: Terraform Validate
    runs-on: ubuntu-latest

    steps:
      - name: Checkout Specific Branch
        uses: actions/checkout@v3
        with:
          ref: ${{ github.ref }}

      - name: Debug Branch Information
        run: |
          echo "Triggered by branch: ${{ github.ref }}"
          echo "Current branch: $(git branch --show-current)"

      - name: Set up Terraform
        uses: hashicorp/setup-terraform@v2

      - name: Validate Terraform Formatting
        id: fmt-check
        run: terraform fmt -check
        working-directory: terraform
        continue-on-error: true  # Continue even if there are formatting issues

      - name: Fix Terraform Formatting Issues
        if: failure()  # Run only if the previous step fails
        run: terraform fmt
        working-directory: terraform

      - name: Terraform Init
        id: init
        run: terraform init
        working-directory: terraform 

      - name: Terraform Validate
        id: validate
        run: terraform validate
        working-directory: terraform

Key Components:

Triggers:
- Push events to infra_features branch
- Manual trigger via workflow_dispatch
- Path filters for terraform directory changes
Environment Setup:
- AWS credentials configuration
- Region specification from variables
Validation Steps:
- Checkout code
- Setup Terraform
- Check and fix Terraform formatting
- Initialize and validate Terraform configurations

The workflow ensures:

Early detection of configuration errors
Consistent code formatting
Basic syntax and configuration checks
Automatic format fixing when issues are found
Immediate feedback to developers

terraform-plan.yml

This workflow handles infrastructure planning and cost estimation. It exists in both main and infra_main branches to provide cost insights before any infrastructure changes are approved.

name: Terraform Plan and Cost Estimation
on:
  workflow_dispatch:
  pull_request:
    branches:
      - infra_main
    types:
      - opened
      - synchronize
      - reopened
permissions:
  pull-requests: write

env:
  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
  TF_VAR_aws_region: ${{ vars.AWS_REGION }}


  TF_VAR_ami_id: ${{ vars.AMI_ID }}
  TF_VAR_instance_type: ${{ vars.INSTANCE_TYPE }}
  TF_VAR_key_pair_name: ${{ vars.KEY_PAIR_NAME }}
  TF_VAR_instance_name: ${{ vars.INSTANCE_NAME }}
  TF_VAR_domain_name: ${{ vars.DOMAIN_NAME }}
  TF_VAR_frontend_domain: ${{ vars.FRONTEND_DOMAIN }}
  TF_VAR_db_domain: ${{ vars.DB_DOMAIN }}
  TF_VAR_traefik_domain: ${{ vars.TRAEFIK_DOMAIN }}
  TF_VAR_grafana_domain: ${{ vars.GRAFANA_DOMAIN }}
  TF_VAR_prometheus_domain: ${{ vars.PROMETHEUS_DOMAIN }}
  TF_VAR_cert_email: ${{ vars.CERT_EMAIL }}
  TF_VAR_private_key_path: ${{ vars.PRIVATE_KEY_PATH }}
  TF_VAR_app_dir: ${{ vars.APP_DIR }}
  TF_VAR_repo: ${{ vars.REPO }}

jobs:
  terraform-plan:
    name: Terraform Plan and Cost Estimation
    runs-on: ubuntu-latest

    steps:
      - name: Checkout PR Branch
        uses: actions/checkout@v3
        with:
          ref: ${{ github.head_ref }}

      - name: Checkout Base Branch
        uses: actions/checkout@v3
        with:
          ref: ${{ github.base_ref }}
          path: base-branch

      - name: Debug Branch Information
        run: |
          echo "Base branch: ${{ github.base_ref }}"
          echo "Head branch: ${{ github.head_ref }}"

      - name: Prepare Terraform
        uses: hashicorp/setup-terraform@v2

      - name: Initialize Terraform Configuration
        run: terraform init
        working-directory: terraform

      - name: Generate Terraform Plan
        id: tf_plan
        run: |
          terraform plan -out=tfplan -lock=false
          terraform show -no-color tfplan > /tmp/plan_output.txt
        working-directory: terraform

      - name: Install Cost Analysis Tool
        uses: infracost/actions/setup@v2
        with:
          api-key: ${{ secrets.INFRACOST_API_KEY }}

      - name: Perform Base Branch Cost Breakdown
        run: |
          cd base-branch/terraform
          infracost breakdown --path=. --format=json > /tmp/base_cost_analysis.json
        continue-on-error: true

      - name: Perform Current Branch Cost Breakdown
        run: |
          cd terraform
          infracost breakdown --path=. --format=json > /tmp/current_cost_analysis.json
          infracost breakdown --path=. --format=table > /tmp/cost_summary.txt

      - name: Generate Cost Difference
        run: |
          infracost diff \
            --path=terraform \
            --compare-to=/tmp/base_cost_analysis.json \
            --format=json > /tmp/cost_difference.json || true
        continue-on-error: true

      - name: Prepare Workflow Report
        uses: actions/github-script@v6
        if: github.event_name == 'pull_request'
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          script: |
            const fs = require('fs');

            const planContent = fs.readFileSync('/tmp/plan_output.txt', 'utf8');
            const costSummary = fs.readFileSync('/tmp/cost_summary.txt', 'utf8');

            const commentBody = `### 🔍 Infrastructure Validation Report

            <details>
            <summary>📋 Terraform Plan Insights 📋</summary>

            \`\`\`terraform
            ${planContent}
            \`\`\`
            </details>

            <details>
            <summary>💰 Cost Estimation Overview 💰</summary>

            \`\`\`
            ${costSummary}
            \`\`\`
            </details>

            *Analysis triggered by @${{ github.actor }}*`;

            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: commentBody
            });

Key Components:

Triggers:
- Pull requests to infra_main branch
- PR events: opened, synchronized, reopened
- Manual workflow dispatch for testing
Environment Setup:
- AWS credentials from secrets
- Comprehensive Terraform variables including:
  - Infrastructure settings
  - Domain configurations
  - Application paths
  - Service endpoints
Planning Process:
- Format validation
- Base branch comparison
- Plan generation
- Cost analysis via Infracost
PR Integration:
- Automated PR comments with:
  - Infrastructure changes
  - Cost estimations
  - Resource modifications
  - Base vs proposed comparisons

The workflow ensures:

Detailed cost breakdowns
Change cost implications
Resource-specific pricing
Cost comparison with current state
Clear plan presentation in PR comments

Important: The workflow requires an Infracost API key stored in GitHub secrets for cost estimation features to work properly.

terraform-apply.yml

This workflow is responsible for actual infrastructure deployment. It exists in both main and infra_main branches to ensure proper triggering on PR merges and to allow manual infrastructure management.

name: Terraform Infrastructure Apply
on:
  pull_request:
    branches:
      - 'infra_main'  
    types:
      - closed  

  workflow_dispatch:
    inputs:
      action:
        type: choice
        description: 'Select the action to perform'
        required: true
        default: 'destroy'
        options:
          - 'destroy'
          - 'apply'

env:
  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
  TF_VAR_aws_region: ${{ vars.AWS_REGION }}

# Terraform Variables to be passed as environment variables
  TF_VAR_ami_id: ${{ vars.AMI_ID }}
  TF_VAR_instance_type: ${{ vars.INSTANCE_TYPE }}
  TF_VAR_key_pair_name: ${{ vars.KEY_PAIR_NAME }}
  TF_VAR_instance_name: ${{ vars.INSTANCE_NAME }}
  TF_VAR_domain_name: ${{ vars.DOMAIN_NAME }}
  TF_VAR_frontend_domain: ${{ vars.FRONTEND_DOMAIN }}
  TF_VAR_db_domain: ${{ vars.DB_DOMAIN }}
  TF_VAR_traefik_domain: ${{ vars.TRAEFIK_DOMAIN }}
  TF_VAR_grafana_domain: ${{ vars.GRAFANA_DOMAIN }}
  TF_VAR_prometheus_domain: ${{ vars.PROMETHEUS_DOMAIN }}
  TF_VAR_cert_email: ${{ vars.CERT_EMAIL }}
  TF_VAR_private_key_path: ${{ vars.PRIVATE_KEY_PATH }}
  TF_VAR_app_dir: ${{ vars.APP_DIR }}
  TF_VAR_repo: ${{ vars.REPO }}

jobs:
  terraform-apply:
    name: Terraform Infrastructure Apply
    runs-on: ubuntu-latest

    # Trigger on merged PR to infra_main or manual destroy
    if: >
      (github.event_name == 'pull_request' && 
       github.event.pull_request.merged == true && 
       github.base_ref == 'infra_main') ||
      (github.event_name == 'workflow_dispatch')

    steps:
      - name: Checkout PR Branch
        uses: actions/checkout@v3
        with:
          ref: ${{ github.head_ref }}

      - name: Install Terraform
        uses: hashicorp/setup-terraform@v2

      - name: Initialize Terraform Configuration
        run: terraform init
        working-directory: terraform

      - name: Terraform Apply
        if: |
          github.event_name == 'pull_request' && 
          github.event.pull_request.merged == true || 
          (github.event_name == 'workflow_dispatch' && github.event.inputs.action == 'apply')
        run: terraform apply --auto-approve -lock=false
        working-directory: terraform

      - name: Terraform Destroy
        if: github.event_name == 'workflow_dispatch' && github.event.inputs.action == 'destroy'
        run: terraform destroy --auto-approve -lock=false
        working-directory: terraform

      - name: Create Action Indicator
        run: echo ${{ github.event.inputs.action }} > ./ansible/action_indicator.txt

      # Upload inventory, trigger  and key as artifacts
      - name: Upload Inventory and Key
        if: github.event_name != 'workflow_dispatch' || github.event.inputs.action != 'destroy'
        uses: actions/upload-artifact@v4
        with:
          name: infrastructure-artifacts
          path: |
            ansible/inventory.ini
            ansible/action_indicator.txt
            terraform/${{ vars.KEY_PAIR_NAME }}
          retention-days: 1

      - name: Save Infrastructure Details as Secrets
        if: github.event_name != 'workflow_dispatch' || github.event.inputs.action != 'destroy'
        run: |
          # First check if files exist
          echo "Checking infrastructure files..."
          if [ ! -f "terraform/${{ vars.KEY_PAIR_NAME }}" ] || [ ! -f "ansible/inventory.ini" ]; then
            echo "Required files not found!"
            exit 1
          fi

          echo "Reading infrastructure files..."
          # Read files and encode in base64 to handle multiline content safely
          SSH_KEY=$(cat "terraform/${{ vars.KEY_PAIR_NAME }}" | base64 -w 0)
          INVENTORY=$(cat "ansible/inventory.ini" | base64 -w 0)

          echo "Saving to GitHub Secrets..."
          # Save SSH key
          gh secret set EC2_SSH_KEY --body "$SSH_KEY"
          # Save inventory
          gh secret set EC2_INVENTORY --body "$INVENTORY"

          echo "Infrastructure details saved as secrets successfully!"
        env:
          GITHUB_TOKEN: ${{ secrets.PAT_TOKEN }}
          GH_TOKEN: ${{ secrets.PAT_TOKEN }}

Key Components:

Triggers:
- PR merge to infra_main branch
- Manual workflow dispatch with options:
  - Apply: For manual infrastructure deployment
  - Destroy: For infrastructure teardown
Environment Configuration:
- AWS credentials from secrets
- Terraform variables for:
  - AWS region
  - AMI configuration
  - Domain settings
  - Application directories etc
Critical Operations:
- Infrastructure deployment via Terraform apply
- Infrastructure destruction (manual trigger)
- Creation of artifacts for downstream workflows
- Important: Saves infrastructure details as GitHub secrets:
  - EC2_SSH_KEY: Base64 encoded SSH key
  - EC2_INVENTORY: Base64 encoded inventory file
  - These secrets are crucial for CD-application workflow

Note: The saving of infrastructure details as secrets is crucial for the CD-application workflow which we'll cover later. These secrets enable secure access to the deployed infrastructure.

ansible-monitoring.yml and Associated Playbook

This workflow automates our monitoring stack deployment, being triggered automatically after successful infrastructure provisioning. The workflow exists in both main and infra_main branches for proper functionality.

name: Monitoring Stack Deployment

on:
  workflow_run:
    workflows: ["Terraform Infrastructure Apply"]
    types:
      - completed

jobs:
  ansible-monitoring:
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    name: Monitoring Stack Deployment
    runs-on: ubuntu-latest

    steps:
      - name: Checkout infra_main Branch
        uses: actions/checkout@v4
        with:
          ref: infra_main  # Explicitly checkout infra_main branch
          fetch-depth: 0   # Get full history

      - name: Download artifacts
        uses: dawidd6/action-download-artifact@v3
        with:
          workflow: terraform-apply.yml
          workflow_conclusion: success
          name: infrastructure-artifacts
          path: ./artifacts

      - name: Check Action Indicator
        id: check-action
        run: |
          ACTION=$(cat ./artifacts/ansible/action_indicator.txt)
          echo "Action: $ACTION"
          if [ "$ACTION" != "apply" ]; then
            echo "Monitoring stack deployment skipped due to action: $ACTION"
            exit 0
          fi

      # Copy the key to the terraform directory
      - name: Setup SSH Key
        run: |
            cp ./artifacts/terraform/${{ vars.KEY_PAIR_NAME }} ./terraform/
            chmod 600 ./terraform/${{ vars.KEY_PAIR_NAME }}

      - name: Update Inventory File
        run: |
          sed -i "s|ansible_ssh_private_key_file=\./${{ vars.KEY_PAIR_NAME }}|ansible_ssh_private_key_file=./terraform/${{ vars.KEY_PAIR_NAME }}|" ./artifacts/ansible/inventory.ini
          cat ./artifacts/ansible/inventory.ini  # Debug print

      - name: Set up Ansible
        uses: alex-oleshkevich/setup-ansible@v1.0.1
        with:
          version: "9.3.0"


      - name: Run Ansible Playbook
        run: |
          ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i ./artifacts/ansible/inventory.ini ./ansible/playbook.yml \
              --extra-vars "domain_name=${{ vars.DOMAIN_NAME }} \
                traefik_domain=${{ vars.TRAEFIK_DOMAIN }} \
                cert_email=${{ vars.CERT_EMAIL }}\
                repo=${{ vars.REPO }}\
                app_dir=${{ vars.APP_DIR }}\
                branch=infra_main"

Key Components:

Workflow Triggers:
- Automatic trigger after terraform-apply completion
- Checks action indicator to ensure proper execution should terraform apply workflow was only triggered for destroy
- Only proceeds on 'apply' action
Infrastructure Access:
- Downloads terraform-generated artifacts
- Configures SSH access using infrastructure keys
- Updates inventory file paths and permissions
Monitoring Setup:
- Deploys complete monitoring stack via Ansible
- Configures domains and certificates
- Sets up repository and application directory

Associated Ansible Playbook Overview

main playbook
- name: Setting up application and monitoring servers
  hosts: all
  become: yes
  roles:
    - docker
    - file_setup
    - monitoring_setup
--------
file setup task
- name: Clone the repository
  git:
    repo: "{{ repo }}"
    dest: "{{ app_dir }}"
    version: "{{ branch }}"


- name: Ensure the Traefik directory exists
  file:
    path: "{{ app_dir }}/traefik"
    state: directory
    mode: '0755'

- name: Create acme.json with proper permissions
  file:
    path: "{{ app_dir }}/traefik/acme.json"
    state: touch
    mode: '0600'

- name: Check if the data folder exists
  stat:
    path: "{{ app_dir }}/data"
  register: data_folder

- name: Ensure Loki data folder has the correct ownership and permissions
  block:
    - name: Change ownership of data folder for Loki
      command: chown -R 10001:10001 ./data
      args:
        chdir: "{{ app_dir }}"

    - name: Set permissions for the data folder
      command: chmod -R 755 ./data
      args:
        chdir: "{{ app_dir }}"
  when: data_folder.stat.exists

- name: Configure monitoring compose file using template
  template:
    src: "monitoring_stack_template.j2"
    dest: "{{ app_dir }}/monitoring-stack.yml"
    mode: '0644'
-----
the monitoring task
- name: Bring up the monitoring stack
  command: docker compose -f monitoring-stack.yml up -d
  args:
    chdir: "{{ app_dir }}"

The playbook orchestrates the entire monitoring setup through three distinct roles:

Docker Role:
- Ensures Docker is installed and configured
- Sets up required Docker services
- Configures networking prerequisites
File Setup Role:
- Clones configuration repository
- Creates necessary directory structure
- Sets up Traefik and SSL configurations
- Manages permissions and ownership
- Prepares data persistence directories
Monitoring Setup Role:
- Deploys the complete monitoring stack
- Configures Prometheus, Grafana, and Loki
- Sets up reverse proxy with Traefik
- Ensures all services are running

The workflow ensures:

Setup Integrity:
- Proper directory structure
- Correct file permissions
- Required configurations
Security Measures:
- Secure credential handling
- Protected certificates
- Proper file permissions
Monitoring Components:
- Prometheus for metrics
- Grafana for visualization
- Loki for logs
- Traefik for routing

Note: This workflow is dependent on the artifacts generated by terraform-apply.yml, showcasing the workflow dependencies in our infrastructure pipeline and it is located both in the main branch and the infra_main branch.

ci-application.yml

This workflow manages the continuous integration process for our application, building and pushing Docker images while updating deployment configurations(compose file).

name: Application CI Pipeline

on:

  push:
    branches:
      - 'integration'
    paths:
      - 'backend/**'
      - 'frontend/**'

env:
  DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
  DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
  IMAGE_TAG: 1.0.${{ github.run_number }}
  FRONTEND_IMAGE: frontend-dojodevops
  BACKEND_IMAGE: backend-dojodevops

jobs:
  build-and-push:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v4
        with:
          token: ${{ secrets.GITHUB_TOKEN }} 
      - name: Login to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Build and push backend
        uses: docker/build-push-action@v5
        with:
          context: ./backend
          push: true
          tags: |
            ${{ secrets.DOCKERHUB_USERNAME }}/${{ env.BACKEND_IMAGE }}:${{ env.IMAGE_TAG }}
            ${{ secrets.DOCKERHUB_USERNAME }}/${{ env.BACKEND_IMAGE }}:latest

      - name: Build and push frontend
        uses: docker/build-push-action@v5
        with:
          context: ./frontend
          push: true
          tags: |
            ${{ secrets.DOCKERHUB_USERNAME }}/${{ env.FRONTEND_IMAGE }}:${{ env.IMAGE_TAG }}
            ${{ secrets.DOCKERHUB_USERNAME }}/${{ env.FRONTEND_IMAGE }}:latest


      - name: Update docker-compose.yml
        run: |
          echo "=== Current docker-compose template content ==="
          cat ./ansible/roles/app_deploy/templates/docker-compose.yml.j2

          echo -e "\n=== Attempting to update image tags ==="
          # Update backend image
          sed -i "s|image: ${{ secrets.DOCKERHUB_USERNAME }}/${{ env.BACKEND_IMAGE }}:.*|image: ${{ secrets.DOCKERHUB_USERNAME }}/${{ env.BACKEND_IMAGE }}:${{ env.IMAGE_TAG }}|" ./ansible/roles/app_deploy/templates/docker-compose.yml.j2
          # Update frontend image
          sed -i "s|image: ${{ secrets.DOCKERHUB_USERNAME }}/${{ env.FRONTEND_IMAGE }}:.*|image: ${{ secrets.DOCKERHUB_USERNAME }}/${{ env.FRONTEND_IMAGE }}:${{ env.IMAGE_TAG }}|" ./ansible/roles/app_deploy/templates/docker-compose.yml.j2

          echo -e "\n=== Updated docker-compose template content ==="
          cat ./ansible/roles/app_deploy/templates/docker-compose.yml.j2

          # Check if any changes were made
          if git diff --quiet ./ansible/roles/app_deploy/templates/docker-compose.yml.j2; then
            echo "No changes were made to docker-compose template"
            echo "Current content of docker-compose template:"
            cat ./ansible/roles/app_deploy/templates/docker-compose.yml.j2
            exit 1
          else
            echo "Changes detected in docker-compose template:"
            git diff ./ansible/roles/app_deploy/templates/docker-compose.yml.j2
          fi

      # Using GitHub's default token for authentication
      - name: Commit and push updated docker-compose
        run: |
          git config --global user.name "${{ github.actor }}"
          git config --global user.email "${{ github.actor }}@users.noreply.github.com"
          git add ./ansible/roles/app_deploy/templates/docker-compose.yml.j2
          git commit -m "Update image tags to ${{ env.IMAGE_TAG }}"
          git push origin integration

Key Components:

Triggers:
- Push events to integration branch
- Path filters for backend and frontend directories
Environment Setup:
- Docker Hub credentials
- Image versioning using GitHub run number
- Separate images for frontend and backend
Build Process:
- Docker Buildx setup for multi-platform builds
- Parallel builds for frontend and backend
- Tag management with latest and versioned tags
- Docker compose template updates

The workflow ensures:

Automated image builds
Version control of images
Configuration updates
Proper image tagging
Automated commit of updated configurations

cd-application.yml

This workflow handles the continuous deployment of our application, utilizing infrastructure secrets from previous workflows.

name: Application CD Pipeline

on:
  workflow_dispatch:
  pull_request:
    branches:
      - 'deployment'
    types:
      - closed

jobs:
  application-deploy:
    if: github.event.pull_request.merged == true
    runs-on: ubuntu-latest
    steps:
      - name: Checkout deployment branch
        uses: actions/checkout@v4
        with:
          ref: deployment

      - name: Setup Infrastructure Files
        run: |
          # Create directories
          mkdir -p ./tmp/ ./tmp/ansible

          # Decode and save SSH key
          echo "${{ secrets.EC2_SSH_KEY }}" | base64 -d > ./tmp/${{ vars.KEY_PAIR_NAME }}
          chmod 600 ./tmp/${{ vars.KEY_PAIR_NAME }}

          # Decode and save inventory
          echo "${{ secrets.EC2_INVENTORY }}" | base64 -d > ./tmp/ansible/inventory.ini

      - name: Update Inventory File
        run: |
          sed -i "s|ansible_ssh_private_key_file=\./${{ vars.KEY_PAIR_NAME }}|ansible_ssh_private_key_file=./tmp/${{ vars.KEY_PAIR_NAME }}|" ./tmp/ansible/inventory.ini

      - name: Set up Ansible
        uses: alex-oleshkevich/setup-ansible@v1.0.1
        with:
          version: "9.3.0"

      - name: Run Application Deployment Playbook
        run: |
          ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i ./tmp/ansible/inventory.ini ./ansible/playbook.yml \
            --extra-vars '{
              "repo": "${{ vars.REPO }}",
              "app_dir": "${{ vars.APP_DIR }}",
              "backend_env": ${{ toJSON(secrets.BACKEND_ENV) }},
              "frontend_env": ${{ toJSON(secrets.FRONTEND_ENV) }},
              "frontend_domain": "${{ vars.FRONTEND_DOMAIN }}",
              "DOCKERHUB_USERNAME": "${{ secrets.DOCKERHUB_USERNAME }}",
              "FRONTEND_IMAGE": "frontend-dojodevops",
              "BACKEND_IMAGE": "backend-dojodevops",
              "db_domain": "${{ vars.DB_DOMAIN }}",
              "branch": "deployment"
            }'

Key Components:

Triggers:
- Pull request merge to deployment branch
- Manual workflow dispatch
Infrastructure Setup:
- Utilizes secrets from terraform-apply:
  - EC2_SSH_KEY for server access
  - EC2_INVENTORY for server details
Deployment Process:
- Ansible configuration
- Environment setup
- Application deployment

The workflow ensures:

Secure deployment process
Environment configuration
Infrastructure access
Application setup by triggering the ansible playbook

Important Environment Variables and Secrets Guide:

Infrastructure Secrets (Generated by terraform-apply):

EC2_SSH_KEY: Base64 encoded SSH key for EC2 access

EC2_INVENTORY: Base64 encoded Ansible inventory file

Docker Hub Credentials:

DOCKERHUB_USERNAME: Docker Hub account username

DOCKERHUB_TOKEN: Docker Hub access token

Application Environment Variables:

BACKEND_ENV: JSON object containing backend environment variables

FRONTEND_ENV: JSON object containing frontend environment variables

Domain Configuration:

FRONTEND_DOMAIN: Domain for frontend service

DB_DOMAIN: Domain for database service

Repository Settings:

REPO: Repository URL

APP_DIR: Application directory path

KEY_PAIR_NAME: Name of the SSH key pair

Note: The CD workflow showcases the importance of proper secret management, using infrastructure secrets generated during provisioning for secure deployment access.

Clean Up

This can be done by manually triggering terraform destroy.

DEV Community

Implementing a Complete GitOps Pipeline: Infrastructure Provisioning and Application Deployment Automation

Introduction

Architecture Overview

Project Overview

Core Components and Workflow Structure

Workflow Triggers and Placement

Expected Outcomes

Infrastructure Configuration Pipeline Deep Dive

terraform-validate.yml

Key Components:

The workflow ensures:

terraform-plan.yml

Key Components:

The workflow ensures:

terraform-apply.yml

Key Components:

ansible-monitoring.yml and Associated Playbook

Key Components:

Associated Ansible Playbook Overview

The workflow ensures:

ci-application.yml

Key Components:

The workflow ensures:

cd-application.yml

Key Components:

The workflow ensures:

Clean Up

Top comments (0)

Read next

Introduction to Amazon Simple Notification Service (SNS)

Matt Garman keynote takeways

Full-Stack Web Application with AWS Amplify: AWS Project

Mastering CI/CD: How to Build a Robust Pipeline with GitHub Actions, Docker, and ECS