DEV Community

Cover image for JavaScript monorepos with Lerna
Semaphore profile image Tomas Fernandez
Tomas Fernandez for Semaphore

Posted on • Originally published at semaphoreci.com

JavaScript monorepos with Lerna

#monorepo #javascript #beginners #cicd

It's no secret that code sharing speeds up development. And there is no better way of teaming up and collaborating than with a monorepo — provided you have the right tools to work with it.

What is Lerna

Lerna is a monorepo manager for JavaScript projects. It helps you take a large codebase and split it into independently deployable packages. Lerna handles every step in the release process — from downloading dependencies, linking packages together, to testing and publishing updated packages to the NPM registry.

By running on top of traditional tools such as npm and Yarn, Lerna can understand how the packages in the repository are interconnected. Needless to say, this makes it so easy to cross-reference and link packages in the same repository.

Image description

Who is using Lerna

You don't have to take my word on it. Lerna is an integral part of the development cycle of incredibly popular projects such as Babel, Facebook's Jest, Gatsby, Google's AngularJS, EmberJS, and MeteorJS.

Image description

Versioning modes in Lerna

Before using Lerna you need to decide on a versioning scheme for your repository. Lerna supports two modes: fixed and independent.

In fixed mode, Lerna maintains the same version for every package in the repository. Updated packages will always be bumped to the same version together. This is the default mode.

Independent mode means that each package is versioned separately, allowing maintainers to bump versions independently. On publication, you'll be prompted on what to do with each updated package.

Creating a new monorepo

We have a little JavaScript monorepo demo to play with here:

GitHub logo TomFern / semaphore-demo-monorepo-javascript

Demo project to try out Yarn workspaces and Lerna features for monorepos.

JavaScript Monorepo Demo

Build Status

This demo repository teaches you how to use Yarn Workspaces and Lerna to manage JavaScript monorepos. You’ll build a monorepo from two separate applications.

Check the final branch to see the end result.

Contents

This repository contains two projects. Their folders are:

  • api: An API that returns geographical information.
  • web: A static website generated from the information returned by the API.

You can run each project independently or in combination. Check each folder to learn how to run them piecemeal. The next section shows how to run them together.

Setup monorepo with Yarn workspaces

  1. Fork this repo.
  2. Clone it to your machine.
  3. Download latest yarn version 
    $ yarn set version berry
    Enter fullscreen mode Exit fullscreen mode
  4. Initialize top-level package.json 
    $ yarn init -w
    Enter fullscreen mode Exit fullscreen mode
  5. Move projects to their workspaces 
    $ git mv api web packages
$ yarn workspaces list
    Enter fullscreen mode Exit fullscreen mode
  6. Install all modules 
    $ yarn install
    Enter fullscreen mode Exit fullscreen mode
  7. Delete old yarn.lock 
    $
    …
    Enter fullscreen mode Exit fullscreen mode
View on GitHub

Feel free to fork it and clone it as you follow this tutorial.

We'll start by generating a Lerna config with lerna init.

$ lerna init
Creating package.json
Creating lerna.json
Creating packages directory
Initialized Lerna files
Enter fullscreen mode Exit fullscreen mode

Move all your applications, libraries, subprojects, and shared code into the packages folder. Each project should have a package.json and, ideally, a lockfile.

$ lerna import api
$ lerna import web
Enter fullscreen mode Exit fullscreen mode

Lerna should now be detecting the packages. Which, for the demo, are two: a GraphQL API service and a Next.js static website.

$ lerna ls
api
web
found 2 packages
Enter fullscreen mode Exit fullscreen mode

Use lerna bootstrap to download NPM dependencies and cross-link packages in the repository.

$ lerna bootstrap
Enter fullscreen mode Exit fullscreen mode

Now you should be able to run all tests found in every package with lerna run. Try them to make sure they work well as a group — the demo ships with unit and integration tests.

$ lerna exec npm run lint
$ lerna exec npm dev &
$ lerna exec npm test
$ lerna exec npm run test integration
Enter fullscreen mode Exit fullscreen mode

Pre-release checks

We're going to publish the packages to npmjs.com. To try out this part, you'll need at least a free account on the service. Once logged in, generate an automation token and copy the value shown somewhere safe. We'll need it in a few minutes.

While you're at it, if you haven't already, authenticate your machine with npm login.

NPM requires that all packages have unique identifiers, so we can't use the names that came with the demo repository. Hence, rename the packages by editing their respective packages.json.

Probably the easiest way of making the package name unique is by scoping them. You can make a package scoped by prefixing the name with your NPM username. In my case, I would change the first few lines of packages.json like this:

  "name": "@tomfern/api",
  "publishConfig": {
    "access": "public"
  }
Enter fullscreen mode Exit fullscreen mode

Commit the changes to the Git repository is clean:

$ git add lerna.json package.json packages
$ git commit -m "install lerna, ready to publish"
Enter fullscreen mode Exit fullscreen mode

Publishing your packages

Publishing a package is a two-step process. First, Lerna pushes all the changes to the remote repository and creates a Git tag. Then, it deploys the updates to NPM. Lerna uses Git tags to mark releases and track changes.

The first step is accomplished with lerna version.

$ lerna version
? Select a new version (currently 0.0.0) (Use arrow keys)
  Patch (0.0.1)
  Minor (0.1.0)
  Major (1.0.0)
  Prepatch (0.0.1-alpha.0)
  Preminor (0.1.0-alpha.0)
  Premajor (1.0.0-alpha.0)
  Custom Prerelease
  Custom Version
Enter fullscreen mode Exit fullscreen mode

Lerna wants to know what the following version number should be. Using semantic versioning, we have to decide how to number this release:

  • patch (1.2.X): when it doesn't introduce behavior changes. For example, to fix a bug.
  • minor (1.X.3): when the version includes backward-compatible changes.
  • major (X.2.3): when version introduces breaking changes.

Before making the change, Lerna will ask for confirmation:

Changes:
 - @tomfern/api: 1.0.0. => 1.2.3
 - @tomfern/web: 1.0.0 => 1.2.3

? Are you sure you want to create these versions?
Enter fullscreen mode Exit fullscreen mode

After picking a version, Lerna creates a tag and pushes it:

$ lerna publish from-git
Found 2 packages to publish:
 - @tomfern/api => 1.2.3
 - @tomfern/web => 1.2.3
? Are you sure you want to publish these packages?
Enter fullscreen mode Exit fullscreen mode

You can also combine versioning and publishing in one command:

$ lerna publish patch
Enter fullscreen mode Exit fullscreen mode

Change-detection

Lerna understands Git and JavaScript. Therefore, it can detect when a package has changed. To try it, change a file and run ~lerna changed~.

$ lerna changed
Looking for changed packages since v1.2.3
@tomfern/api
found 1 package ready to publish
Enter fullscreen mode Exit fullscreen mode

You can find per-package changes details with lerna diff.

Try publishing the updated version by re-running lerna version and lerna publish.

Configuring the CI/CD pipeline

For this part you'll need a Semaphore account. If you don't have one, you can create a trial account for free with GitHub.

Now the trick is to automate all these processes in a CI/CD pipeline. The plan is to:

  1. Install and cache all dependencies.
  2. Run all tests.
  3. If we are on a tagged release, publish the packages.

After login in into Semaphore, click on create new to add a new project.

Image description

Choose the forked repository.

Image descriptionFinally, select "single job" and click on customize.

Image description

Install job

The build stage bootstraps the repository and caches the downloaded dependencies. We use lerna bootstrap and then npm exec cache to store the contents of node_modules in the Semaphore cache.

npm install --global lerna
checkout
lerna exec -- cache restore node-modules-\$LERNA_PACKAGE_NAME-$SEMAPHORE_GIT_BRANCH,node-modules-\$LERNA_PACKAGE_NAME
lerna bootstrap
lerna exec -- cache store node-modules-\$LERNA_PACKAGE_NAME-$SEMAPHORE_GIT_BRANCH,node-modules-\$LERNA_PACKAGE_NAME node_modules
Enter fullscreen mode Exit fullscreen mode

Image description
Test block

No continuous integration should lack tests. Our demo includes three types of tests:

  • Linter: runs eslint to run static code analysis tests.
  • Unit tests: executes unit tests in all packages.
  • Integration test: executes the integration test suite.

Click on add block and scroll down on the right pane to the prologue. The prologue is executed before any jobs in the block. Type the following commands to retrieve the cached dependencies.

npm install --global lerna
checkout
lerna exec -- cache restore node-modules-\$LERNA_PACKAGE_NAME-$SEMAPHORE_GIT_BRANCH,node-modules-\$LERNA_PACKAGE_NAME
lerna bootstrap
Enter fullscreen mode Exit fullscreen mode

The test jobs are all one-liners. This is the linter:

lerna run lint
Enter fullscreen mode Exit fullscreen mode

Create two more jobs in the block, one for the unit tests:

lerna run test
Enter fullscreen mode Exit fullscreen mode

And one for the integration tests:

lerna run test-integration
Enter fullscreen mode Exit fullscreen mode

Image description

Click on "run the workflow" > start to try out the pipeline.

Image description

Image description

Continuous Deployment

The goal here is to publish packages to the NPM registry using continuous delivery.

We'll begin by creating a secret on Semaphore. Click on settings on the main menu.

Image description

Then go to secrets and press create secret. In value, type NPM_TOKEN and fill in the automation token generated earlier. Save the secret.

Image description
Go back to the workflow in Semaphore and click on edit workflow to open the editor.

Click on add promotion to create a second pipeline. Enable the automatic promotion checkbox and type this line, which selects tagged releases:

tag =~ '.*' AND result = 'passed'
Enter fullscreen mode Exit fullscreen mode

Image description
Click on the first job on the delivery pipeline and use the following commands in the job.

npm install --global lerna
checkout
echo "//registry.npmjs.org/:_authToken=$NPM_TOKEN" > .npmrc
lerna exec -- cache restore node-modules-\$LERNA_PACKAGE_NAME-$SEMAPHORE_GIT_BRANCH,node-modules-\$LERNA_PACKAGE_NAME node_modules
lerna bootstrap
lerna publish from-git --no-git-tag-version --no-push --yes
Enter fullscreen mode Exit fullscreen mode

Scroll down and check the NPM secret created earlier.

Image description
Save the pipeline. It will run one more time, but no releases will take place. Next, try updating one of the packages using lerna version from your own machine.

$ git pull origin main
$ lerna version patch
Enter fullscreen mode Exit fullscreen mode

The pipeline should start when Lerna pushes the tagged release.

Image description

Changed-based testing

Lerna detects by itself which packages have changed since the last release and publishes only the new versions. But this feature only works for publishing, not testing.

While Lerna doesn't support change-based testing, Semaphore does. And it's pretty easy to configure. The trick is in the change_in function, which calculates folder and file changes. Let's see how to use it.

To use change_in, you'll need to create separate test paths for each package or group of packages. In other words, you have to edit the jobs in "Test" so they only operate on one of the packages using the --scope option. As an example, this makes the lint job run only on the @tomfern/api package.

lerna run lint --scope @tomfern/api
Enter fullscreen mode Exit fullscreen mode

Repeat the change in the rest of the test jobs.

lerna run test --scope @tomfern/api

lerna run test-integration --scope @tomfern/api
Enter fullscreen mode Exit fullscreen mode

Image description
Now create a second testing block for the other package and make it dependent on the "Bootstrap" block. This time, use --scope to select the other package.

The magic trick comes now. Scroll down until you reach "Skip/Run conditions" and select Run this block when conditions are met. For example, the following condition is triggered when a file changes in the /packages/api folder.

change_in('/packages/api/', { default_branch: 'main'})
Enter fullscreen mode Exit fullscreen mode

Image description
If your repository's default branch is master, you can omit the { default_branch: 'main' } part.

Repeat the same procedure for the web package:

change_in('/packages/web/', { default_branch: 'main'})
Enter fullscreen mode Exit fullscreen mode

Click on Run the workflow to save the setup and try the pipeline. Well used, change detection can significantly speed up pipelines.

Image description

Next steps

As always, there is still some room for improvement. For example, you might want to use Lerna's package hoisting to reduce the size of the node_modules.

Bear in mind that Lerna can team up with Yarn, if you so prefer. You can switch from npm to yarn by adding these lines into lerna.json:

  "npmClient": "yarn",
  "useWorkspaces": true
Enter fullscreen mode Exit fullscreen mode

One of the benefits of this is that we can use Yarn workspaces to avoid using node_modules altogether.

That's it

Monorepos are gaining popularity. In great part, thanks to improved tooling support. If you have many JavaScript packages in one repository and want to publish them to NPM, Lerna is the right tool for the job.

Are you a JavaScript developer? We have a lot of exciting stuff for you:

Top comments (0)

Read next

thinkthroo profile image

Manage your versioning and changelogs using Changesets

thinkThroo -

devops_den profile image

5 Best Reasons You Should Choose Microservices Architecture for Your Project

Devops Den -

i_am_vesh profile image

Automated testing and reporting are essential components of the CI/CD process in Jenkins,

Avesh -

mikeyoung44 profile image

New Research Breaks Through AI Language Model Safeguards, Exposing Security Risks

Mike Young -