Setting up CI for microservices in monorepo using GitHub Actions

In this article, I'll describe how to set up Continuous Integration for an example application consisting of multiple microservices structured as monorepo and written in Go.

What is monorepo?

In the context of microservices, structuring an application as a monorepo means having a single repository for all microservices with each microservice residing in its own separate directory.

Why monorepo?

To be clear, monorepo isn't a silver bullet for microservices structuring, having its own benefits and downsides. Generally, benefits include simplicity, consistency, stability, and code reuse, while downsides are considered to be tight coupling, git repo scalability, and lack of code access control. I will dive more deeply into the benefits and downsides of structuring microservices as monorepo in a separate article.

Example monorepo

Let's take a look at an example application structured as monorepo. This is a simple HTTP API that consists of three routes: auth, users, and articles.

monorepo-actions-ci
└── routes
    ├── articles
    │   └── main.go
    ├── auth
    │   └── main.go
    └── users
        └── main.go

The important detail is that each route is an independent application with its own entrypoint, main.go.

package main

import "github.com/gin-gonic/gin"

func main() {
    router := gin.Default()

    router.GET("/auth", func(c *gin.Context) {
        c.String(200, "auth")
    })

    router.Run()
}

package main

import "github.com/gin-gonic/gin"

func main() {
    router := gin.Default()

    router.GET("/users", func(c *gin.Context) {
        c.String(200, "users")
    })

    router.Run()
}

package main

import "github.com/gin-gonic/gin"

func main() {
    router := gin.Default()

    router.GET("/articles", func(c *gin.Context) {
        c.String(200, "articles")
    })

    router.Run()
}

Setting up GitHub Actions

Let's create a simple workflow for the auth route, .github/workflows/auth.yaml.

name: "auth"

on:
  push:
    # run the workflow only on changes
    # to the auth route and auth workflow
    paths:
      - "routes/auth/**"
      - ".github/workflows/auth.yaml"

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1

      # run tests for route
      - name: Run tests
        run: |-
          cd routes/auth
          go test ./...

This workflow runs go test when changes are made to the auth route. Additionally, it runs when changes are made to the workflow file, which is useful for debugging the workflow itself.

Automating workflow generation

Now, the next thing would be to add the same workflow for routes/users and routes/articles. However, while in this case just duplicating the workflow seems a reasonable approach, in case of a big application with a complex workflow and a large number of routes this becomes problematic.

Instead, it would be better to automatically generate workflows from a single workflow template. Let's determine which parts of the workflow need to be adjusted when we move from one endpoint to another.

name: "{{ROUTE}}"

on:
  push:
    paths:
      - "routes/{{ROUTE}}/**"
      - ".github/workflows/{{ROUTE}}.yaml"

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1

      - name: Run tests
        run: |-
          cd routes/{{ROUTE}}
          go test ./...

Looks simple. Let's put this template into .github/workflow-template.yaml and create a bash script that generates workflows for all endpoints from this template. Let's put that script into workflows.sh:

# read the workflow template
WORKFLOW_TEMPLATE=$(cat .github/workflow-template.yaml)

# iterate each route in routes directory
for ROUTE in $(ls routes); do
    echo "generating workflow for routes/${ROUTE}"

    # replace template route placeholder with route name
    WORKFLOW=$(echo "${WORKFLOW_TEMPLATE}" | sed "s/{{ROUTE}}/${ROUTE}/g")

    # save workflow to .github/workflows/{ROUTE}
    echo "${WORKFLOW}" > .github/workflows/${ROUTE}.yaml
done

Don't forget to make the script executable after creating the file:

chmod +x ./workflows.sh

Now let's run the script and see what happens:

monorepo-actions-ci
❯ ./workflows.sh
generating workflow for routes/articles
generating workflow for routes/auth
generating workflow for routes/users

After taking a quick look at .github/workflows you'll see that the script automatically generated a workflow for each route present in /routes.

├── .github
│   ├── workflow-template.yaml
│   └── workflows
│       ├── articles.yaml
│       ├── auth.yaml
│       └── users.yaml
├── routes
│   ├── articles
│   │   └── main.go
│   ├── auth
│   │   └── main.go
│   └── users
│       └── main.go
└── workflows.sh

name: "auth"

on:
  push:
    paths:
      - "routes/auth/**"
      - ".github/workflows/auth.yaml"

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1

      - name: Run tests
        run: |-
          cd routes/auth
          go test ./...

name: "users"

on:
  push:
    paths:
      - "routes/users/**"
      - ".github/workflows/users.yaml"

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1

      - name: Run tests
        run: |-
          cd routes/users
          go test ./...

name: "articles"

on:
  push:
    paths:
      - "routes/articles/**"
      - ".github/workflows/articles.yaml"

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1

      - name: Run tests
        run: |-
          cd routes/articles
          go test ./...

Does it work?

Let's push our code to GitHub repository and see what happens.

_{Workflows are starting:}

_{Workflows have finished running successfully:}

_{Auth workflow details:}

Each of the workflows has successfully finished running. Success! 🎉

I hope this article will come handy to you one day. Let me know your thoughts in the comments below or on Twitter. 🖖