Lerna and Turborepo with Gitlab CI/CD

In this post we're going to cover a simple approach to a monorepo setup with a fully automated CI/CD workflow with Lerna, Turborepo, and Gitlab CI/CD.

Here is the project link to follow along and use for your own monorepo projects.

Tools

This workflow includes the following tools.

• Lerna: for versioning and publishing packages.
• Turborepo: for tasking running.
• Gitlab CI/CD: for our CI/CD server.
• Husky for Git hooks.
• Commitizen: for its interactive commit cli and changelog adapter. We will tie this into one of our Git hooks through husky - they work nicely together.

Typescript and Jest Configurations

One helpful way to manage Typescript and Jest configurations in a monorepo is to define core configurations in the root of the repository, and extend from the core configurations inside of packages to suit individual package needs.

The tsconfig.json file in the root of the repository has some basic configurations.

{
  "compilerOptions": {
    "module": "commonjs",
    "moduleResolution": "node",
    "declaration": true,
    "noImplicitAny": true,
    "removeComments": true,
    "target": "es6",
    "sourceMap": true,
    "strict": true
  },
  "exclude": ["node_modules"]
}

The packages also have their own tsconfig.json files that make use of the "extends" option that will inherit from the root, and any new settings will override root configuration values.

{
  "extends": "../../tsconfig.json",
  "compilerOptions": {
    "outDir": "./lib"
  },
  "include": ["./src"]
}

Similarly for jest we have a root configuration that is extended and overridable in packages.

import type { Config } from "jest";

const config: Config = {
  verbose: true,
  preset: "ts-jest",
  collectCoverage: true,
};

export default config;

import type { Config } from "jest";
import { default as rootConfig } from "../../jest.config";

const config: Config = {
  ...rootConfig,
};

export default config;

Git, Versioning and Publishing Strategy

Let's break down what our strategy is in the diagram above. In this diagram, P1 and P2 stand for package-one and package-two. Since we're adhering to a trunk-based branching strategy, we have one main branch and as short-lived as possible working branches.

The main branch is where all stable versions of packages will be versioned and published.

Any commits to branches named anything other than main (working branches) will publish canary versions of packages if they have related changes in the commits. The git sha is attached to the version to avoid collisions. This gives us "nightly builds" on working branches so that work in progress can be quickly tested by anyone.

# example from lerna docs

lerna publish --canary
# 1.0.0 => 1.0.1-alpha.0+${SHA} of packages changed since the previous commit
# a subsequent canary publish will yield 1.0.1-alpha.1+${SHA}, etc

# this script we use in our root package.json

"publish:canary": "lerna publish --canary --no-git-tag-version --no-push --yes"

Any commits git commit to working branches are met with a Husky prepare-commit-msg hook that triggers Commitizen's interactive CLI that will prompt you with questions to construct a commit message based on Angular's commit message conventions. The hook also skips if running in CI. Gitlab CI/CD has a predefined variable CI that we can use to identify this.

#!/usr/bin/env sh
. "$(dirname -- "$0")/_/husky.sh"

[ -n "$CI" ] && exit 0

exec < /dev/tty && npx cz --hook || true

Keeping commits very specific and short is helpful and important for a number of reasons. Depending on what kind of commit you select in the interactive CLI will determine which stable version of the package ends up getting updated when merging to main branch. Commit messages will also be added to CHANGELOG.md files which are ideal to keep clean for keeping track of and reporting changes to consumers of your packages.

When merging back to the main branch, the pipeline will run lerna version with some additional arguments which will create and push tags for changed packages as well as update, commit, and push version bumps in their package.json files and CHANGELOG.md files. In order for the pipeline to push changes back to the repository, there are some CI/CD specific things we will need to do which are covered in the Gitlab CI/CD Strategy section.

"version:stable": "lerna version --yes"

We also have some settings in lerna.json for the version command that only allows versioning on the main branch and an adherance to conventional commits.

"command": {
  "version": {
    "allowBranch": "main",
    "conventionalCommits": true
  }
},

Since the first versioning pipeline that runs on a merge to main produces a commit and push, another pipeline is triggered which will use lerna publish with additional arguments. The from-git argument publishes packages tagged in the current commit from our versioning pipeline.

"publish:stable": "lerna publish from-git --yes"

Gitlab CI/CD Strategy

While we covered some CI/CD in the Git, Versioning and Publishing section, this section will focus more on tokens, Gitlab repository settings, and the .gitlab-ci.yml file. For this section you'll need to have access to your Gitlab repository settings - if it's your company Gitlab you may not have maintainer access.

Tokens

For this strategy we're going to need two secret tokens:

• $NPM_TOKEN: to publish packages to the NPM registry or this could be your companies internal Nexus repository. In your npmjs account you can generate a new token in Access Tokens -> Generate New Token and give it an Automation type and make sure to copy the secret value. Now in Gitlab, in Settings -> CI/CD -> Variables add a new variable with key of NPM_TOKEN and value of your copied secret. Make sure to check the Masked option so that your token is masked in job logs. We don't want this token to only be exposed to protected branches, otherwise our canary publish on working branches will not work.

• GL_TOKEN: our versioning pipeline on the main branch pushes tags and a commit with package version bumps and changelog updates. In order for our CI server to be able to do this, we need to give it permissions to do so. For free personal Gitlab accounts you can create a personal access token otherwise if project access tokens are available create one with api, read_repository and write_repository permissions. Now as we did the the $NPM_TOKEN, add this as another CI/CD variable called $GL_TOKEN. If it's a project access token you'll need to assign it a role that has access to push to the main branch. Also, in case you wanted to have Lerna create Gitlab releases, your token would need registry read and write access - but that is out of the scope of this strategy.

YAML File

Let's pick apart our project's .gitlab-ci.yml file.

Firstly, we have 1 stage in our pipeline that will always run just 1 job. It may be helpful to minimize the number of stages and jobs in your pipelines to keep complexity low - this is a personal or team preference.

stages:
  - build

Next we have a .prepare configuration that we can extend in all of our jobs, since we will need to do this preparation before-hand. This is a cleaner way to reuse logic rather than have the same preparation script code in every job.

.prepare:
  before_script:
    - git config user.email $GITLAB_USER_EMAIL
    - git config user.name $GITLAB_USER_NAME
    - git remote set-url origin
      "https://gitlab-ci-token:$GL_TOKEN@$CI_SERVER_HOST/$CI_PROJECT_PATH.git"
    - echo "//registry.npmjs.org/:_authToken=$NPM_TOKEN" > .npmrc
    - git checkout $CI_COMMIT_BRANCH

We're specifying these to be run as before_script which will run before our jobs' primary scripts. Since Lerna will be performing Git operations during versioning and publishing, we set git config user.email $GITLAB_USER_EMAIL and git config user.name $GITLAB_USER_NAME. The $GITLAB_USER_EMAIL and $GITLAB_USER_NAME variables are predefined variables that are the email and name of the user who started the job.

Next we set the job token to assume the access of the $GL_TOKEN variable that we generated as either a personal or project access token with elevated repository permissions. The $CI_SERVER_HOST and $CI_PROJECT_PATH variables are also predefined variables that point to the url of the repository itself. See CI/CD job token. With this set, our release job will be able to push commits and tags back to the repository.

Next we're writing to a .npmrc file that we're wanting to publish to the public npm registry registry.npmjs.org and passing our $NPM_TOKEN variable as the authentication token so that we can publish packages. This could also be your company's internal Nexus repository in which case the registry would be different.

Now since Gitlab CI/CD jobs by default check out the repository at the commit that triggered the job in a DETACHED HEAD state, we need to actually checkout the branch itself so we run git checkout $CI_COMMIT_BRANCH, otherwise Lerna will throw an error.

Our first job in our YAML file build-publish-canary will only running on working branches and will publish our per commit nightly builds. It looks like this:

# CANARY PUBLISHING ON WORKING BRANCHES
build-publish-canary:
  stage: build
  extends:
    - .prepare
  script:
    - yarn install --frozen-lockfile
    - yarn build
    - yarn test
    - yarn publish:canary
  rules:
    - if:
        $CI_COMMIT_BRANCH != "main" && $CI_PIPELINE_SOURCE !=
        'merge_request_event' && $CI_COMMIT_TITLE != "Publish"

We're first making this job part of our 1 build stage. Then we're extending that .prepare configuration which will run its scripts before the script we have defined in this job.

The first part of our script is yarn install --frozen-lockfile which will install our dependencies using the exact state of our yarn.lock file that we have checked into the commit that triggered this job. This is generally a good practice for CI so that our lock file doesn't change when in CI. Then we run the build script defined in our root package.json file. This is where turbo comes in to run tasks. The build script looks like this:

"build": "turbo run build",

Turbo will run all of the build scripts defined in all of our monorepo's packages.

We then run the test script also defined in our root package.json in which turbo is also running our test tasks.

"test": "turbo run test",

Finally we run our publish:canary script which publishes to npm a unique package based on our commit.

An important part of this job are the rules that we have set.

• $CI_COMMIT_BRANCH != main: ensures we're only running this job on branches that are not the main trunk branch.
• $CI_PIPELINE_SOURCE != 'merge_request_event': ensures we're not running this job on merge requests.
• $CI_COMMIT_TITLE != Publish: ensures that we do not run this job after Lerna has produced a commit with the title Publish which is what lerna version will do as we see in the next job.

Our second job build-version-stable will run on merges to main branch or on direct commit pushes to main branch. It looks like this:

# STABLE VERSIONING ON MAIN BRANCH
build-version-stable:
  stage: build
  extends:
    - .prepare
  script:
    - yarn install --frozen-lockfile
    - yarn build
    - yarn test
    - yarn version:stable
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_COMMIT_TITLE != "Publish"

We'll skip what's already been explained in the previous job straight to our version:stablescript which we are running after testing. In the Git, Versioning and Publishing we covered that version:stable is going to create and push tags, update package.json versions and changelogs in our packages which will push a commit back to the main branch. That will then trigger our final job that we have defined.

# STABLE PUBLISHING ON MAIN BRANCH
build-publish-stable:
  stage: build
  extends:
    - .prepare
  script:
    - yarn install --frozen-lockfile
    - yarn build
    - yarn test
    - yarn publish:stable
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_COMMIT_TITLE == "Publish"

This job will finally publish the new versions of our packages! 🥳

Tips

• Don't squash commits on merging feature branches to main branch, the changelog will not reflect the changes that were made. It could be helpful to apply a constraint in Gitlab Settings -> Merge Requests -> Squash commits when merging and do not allow this.
• When updating multiple packages at once on a feature branch, if you want canary publishes for all updated packages, you will need to push committed changes related to each package separately as canary publishing will refer to most recent commit on a feature branch to determine which packages to publish.
• Avoid long running working branches. Commit and merge changes back to main as often as possible.
• Work with two branches only, main and working branches. If bugs are found, simply use a working branch and merge fixes to main.
• Communicate with your team on the value of Semantic Versioning, how to use Commitizen, and to be descriptive and intentional about commits.
• If you're working with a team and certain members are experts or lead maintainers of specific packages, using Gitlab's Code Owners feature can be helpful for specifying specific maintainers who can approve merge requests for certain packages. This can be really helpful for splitting up responsibility. Code Owners requires Gitlab premium.