Efficiently Managing Multi-Directory Repositories with GitHub Actions CI/CD Pipeline

Working with repositories that house multiple subdirectories, each representing an independent service deployed on AWS Lambda, presents unique challenges. One of the primary issues is determining which services need deployment based on changes within their respective directories. In this blog, I will share how I tackled this challenge by creating an efficient CI/CD pipeline using GitHub Actions, divided into two key jobs: detecting changes and building and deploying the necessary services.

Challenges with Multi-Directory Repositories

When dealing with a repository structured with multiple subdirectories, each corresponding to an independent service, it becomes crucial to identify changes at a granular level. Each service must be individually assessed for modifications to ensure that only the necessary components are built and deployed. This selective deployment helps optimize the CI/CD process, saving time and resources.

Solution: Dividing the Pipeline into Two Jobs

To address this issue, I designed the CI/CD pipeline with two primary jobs:

1. detect-changes: This job identifies which folders (services) have changes using git diff and stores them in an array.

2. build-and-deploy: This job takes the array generated in the previous step, creates zip packages, and deploys them as Lambda functions.
   Workflow Configuration
   Here is a detailed breakdown of the GitHub Actions workflow configuration:
   

name: lambda package update

on:
    push:
        branches:
            - dev
        paths:
            - "**"

jobs:
    detect-changes:
        runs-on: ubuntu-latest
        outputs:
            services: ${{ steps.set-services.outputs.services }}
        steps:
            - name: Checkout repository
              uses: actions/checkout@v2
              with:
                  fetch-depth: 0

            - name: Set services variable
              id: set-services
              run: |
                  echo "Checking for changes..."
                  services="[]"

                  current_commit=${{ github.sha }}
                  previous_commit=${{ github.event.before }}

                  for service in serviceA serviceB serviceC; do
                    if git diff --name-only ${{github.event.before}} ${{github.sha}} | grep "^${service}/"; then
                      services=$(jq -c --arg service "$service" '. + [$service]' <<<"$services")
                    fi
                  done
                  echo "::set-output name=services::$services"

            - name: Debug services variable
              run: echo ${{ steps.set-services.outputs.services }}

    build-and-deploy:
        needs: detect-changes
        runs-on: ubuntu-latest
        permissions:
            id-token: write
            contents: read
        if: ${{ needs.detect-changes.outputs.services != '[]' }}
        strategy:
            matrix:
                service: ${{ fromJson(needs.detect-changes.outputs.services) }}
        steps:
            - name: Checkout repository
              uses: actions/checkout@v2

            - name: Set up Node.js
              uses: actions/setup-node@v2
              with:
                  node-version: "20"

            - name: Install dependencies
              run: |
                  cd ${{ matrix.service }}
                  npm install --omit=dev

            - name: Create zip package
              run: |
                  cd ${{ matrix.service }}
                  zip -r ../${{ matrix.service }}.zip node_modules src index.js

            - name: Configure AWS Credentials
              uses: aws-actions/configure-aws-credentials@v1
              with:
                  role-to-assume: role-arn
                  aws-region: region

            - name: Deploy to AWS Lambda
              run: |
                  aws lambda update-function-code --function-name ${{ matrix.service }}-${{ github.ref_name }}-api --zip-file fileb://${{ matrix.service }}.zip

Key Components Explained

Detect Changes Job

• Checkout Repository: Checks out the repository to access the code.
• Set Services Variable: Uses git diff to identify which service directories have changes and sets the services variable accordingly. The services array is then output for use in the next job.

Build and Deploy Job

• Conditional Execution: This job only runs if there are changes detected in the previous step.

• Matrix Strategy: Iterates over each service that has changes.

• Install Dependencies and Create Zip Package: Installs the necessary dependencies and creates a zip package for each modified service.

• AWS Deployment: Deploys the zip package to AWS Lambda, updating the function code.

Benefits and Optimization

By implementing this two-step CI/CD pipeline, I achieved a more efficient and optimized deployment process. This setup ensures that only the services with changes are built and deployed, significantly reducing unnecessary builds and deployments. Additionally, using GitHub Actions' matrix strategy allows for parallel processing of multiple services, further speeding up the deployment process.

Conclusion

Managing multi-directory repositories with independent services can be complex, but with a well-structured CI/CD pipeline, you can streamline the process. By detecting changes and conditionally deploying only the necessary services, you can save time and resources, making your development workflow more efficient. This approach can be adapted to various use cases, providing a scalable solution for complex repository structures.