Beginners Guide to Node.js Continuous Integration

In this series of articles, we will set up continuous integration and deployment for nodejs applications on CircleCI platform.

The Workflow

Every time we push code to the repository CircleCI will be notified of the new code, and it will automatically kick off a continuous integration process that will run our unit tests (via Jest) in node v10, v12, and v13.

First, you will need to create an account at circleci.com and follow the instruction to choose the repository you want to use.

Second, you will need to install CircleCI application from the github marketplace. With the application installed CircleCi will integrate with the chosen repository, and make it easier to see what is going on with the tests. Any time you commit to a branch or create a pull request circleCI will run the tests.

Next, we need to create a circleCI configuration file.

CircleCI configuration file

For the circleCI platform to recognize the repository as ready for the integration, we need to have a special configuration file present in the repository. The default location for the file inside the repository is .circleci/config.yml

This file contains the instructions for the CircleCI platform on how to run tests on the repository.

version: 2.1
jobs:
  node-v10:
    docker:
      - image: circleci/node:10
    steps:
      - test
  node-v12:
    docker:
      - image: circleci/node:12
    steps:
      - test
  node-v13:
    docker:
      - image: circleci/node:13
    steps:
      - test
commands:
  test:
    steps:
      - checkout
      - restore_cache:
          key: dependency-cache-{{ checksum "package-lock.json" }}
      - run:
          name: install-dependancies
          command: npm ci
      - save_cache:
          key: dependency-cache-{{ checksum "package-lock.json" }}
          paths:
            - $HOME/.npm
      - run:
          name: unit test
          command: |
            npm run ci:test
workflows:
  version: 2
  build_and_test:
    jobs:
      - node-v10
      - node-v12
      - node-v13

Configuration File explained

The file is divided into three sections: jobs, commands and workflows. First, we will concentrate on the jobs key

Jobs

Jobs are collections of steps that are executed inside the container.

jobs:
  node-v10:
    docker:
      - image: circleci/node:10
    steps:
      - test
  node-v12:
    docker:
      - image: circleci/node:12
    steps:
      - test
  node-v13:
    docker:
      - image: circleci/node:13
    steps:
      - test

In the above code excerpt, we have defined three jobs and named them node-v10 and and node-v12 (names are arbitrary).

Next, we have a docker key which we can use to pass various options for the customization of the Linux container that is going to be created (we are using Linux containers for testing our code, but circleCI can also spin up complete virtual machines: Linux, Windows, and MacOS).

image option defines what container image we are going to use. In this case, we are using the default CircleCI images for different node runtime versions.
You are free to use other images but the default CircleCI images are sufficient in most cases and they come with some essential tools included (git, ssh, tar, ca-certificates, curl, wget)
You can see exactly what is included on the circleCI docker hub page.

Steps

Steps ( when inside a Job ) are executable commands that are executed during a job.

In this case, we have only one step: test (the name is arbitrary)

This step is a command which contains more steps. The reason we have this step as a command is because of the code reuse. As you can see all jobs (node-v10, node-v12, node-v13) are using the same command test, if we were to have only one job we could just take all the steps and put them under the job key.

like this:

jobs:
  node-v12:
    docker:
      - image: circleci/node:12
    steps:
      - checkout
      - restore_cache:
          key: dependency-cache-{{ checksum "package-lock.json" }}
      - run:
          name: install-dependancies
          command: npm ci
      - save_cache:
          key: dependency-cache-{{ checksum "package-lock.json" }}
          paths:
            - $HOME/.npm
      - run:
          name: unit test
          command: |
            npm run ci:test

Test Command

Command is a sequence of instructions to be executed in a job. The main purpose of commands is to enable the reuse of a single command definition across multiple jobs ( e.q. test command in all jobs v10,v12,v13). Notice also that commands have their own steps.

The name of the command is completely arbitrary. The purpose of this command is to:

1. check out the code from the repository.
2. try to use cached node modules so it doesn't need to download the dependencies again (more on that later).
3. install the dependencies.
4. save the installed dependencies in a cache.
5. run the tests.

Steps are run in order, from top to bottom.

• checkout (step 1) This is a special step built into the CircleCI platform which is used to check out the source code from the repository ( link to docs ).

• restore_cache (step 2) another built in step that is used to "restore a previously saved cache" (more on that later) official documentation

  • key a string under which to look for the cache (we could have many more different caches under different keys)

• run (step 3) used for invoking command-line programs. You can invoke any command that is available inside the container. ( link to docs )

  • name - used in the CircleCI UI to easily differentiate from other steps/commands
  • command - the actual command that is going to run (npm ci)

• save_cache (step 4) Generates and stores a cache of a file or directory of files such as dependencies or source code in CircleCI object storage ( link to docs )

• run (step 5) used for invoking command line programs ( link to docs )

  • name - used in the circleCI UI to easily differentiate from other steps/commands
  • command - the actual command that is going to run (npm ci:test)more on that later

Workflows

Workflows are collections of jobs that are executed on every code push.

workflows:
  version: 2
  build_and_test:
    jobs:
      - node-v10
      - node-v12
      - node-v13

Workflows key determines which workflows (which consist of jobs) are going to run and in what order.
Here we are declaring one workflow build_and_test with three jobs node-v10, node-v12 and node-v13.
These jobs are going to run in parallel ( they can also run in sequence or conditionally )
Since we only have one workflow as soon as the new code push to the repository is detected, CircleCI will launch three Linux containers (docker images with different nodejs versions) and run the tests and report back.

Saving and restoring NPM cache

Installing nodejs dependencies from scratch can be a time-consuming process so to speed up the installation process we are going to use one of the recommended caching strategies when working with node modules.

saving the cache

- save_cache:
    key: dependency-cache-{{ checksum "package-lock.json" }}
    paths:
      - $HOME/.npm
- run:
    name: unit test
    command: |
      npm run ci:test

We are saving the contents of .npm directory for later use. This is the directory that is storing global npm cache ( not global modules). This directory is located in the users home directory.

To reliably validate and invalidate the cache we need to know when the cache becomes invalid. To do that we are saving the cache under a key that is going to be different every time the package-lock.json file is changed.
So this line of code {{ checksum "package-lock.json" }} generates a unique string based on the contents of the package-lock.json
So our key is going to look something like this: dependency-cache-4260817695

restoring the cache

- restore_cache:
    key: dependency-cache-{{ checksum "package-lock.json" }}

In the above step, we are trying to restore the cached version of the .npm directory.
Same as in the saving the cache step we are going to calculate the cache key via the package-lock.json file, and if the file hasn't been changed we are going to get the same key (dependency-cache-4260817695). That means that dependencies are the same and we can leverage the cache.

Running the actual tests

- run:
    name: unit test
    command: |
      npm run ci:test

As you can see in the above example we are running the ci:test task from the package.json file. It is always a good option to have different tasks for running the tests locally and in continuous integration.

ci:test is an npm script that is created for the sole purpose of testing the code in a continuous integration environment.

"ci:test": "jest --runInBand --ci"

As you can see, we are using jest testing framework.

• --runInBand

from the docs:

Run all tests serially in the current process, rather than creating a worker pool of child processes that run tests.

By default, Jest detects how many cores your CPU has, and automatically spreads the tests over all cores. This can be a problem when running in docker or virtual machine because sometimes Jest will not get the correct number when querying for number of cores and you will get "out of memory error"

Note that you can also use another option to limit the number of workers --maxWorkers=<num> This can yield faster tests, but you need to know exactly how many CPU cores you have in your testing environment.

In the case of CircleCI free accounts get their medium tier with 2vCPUs and 4GB of RAM.

• --ci from the docs:

When this option is provided, Jest will assume it is running in a CI environment. This changes the behavior when a new snapshot is encountered. Instead of the regular behavior of storing a new snapshot automatically, it will fail the test and require Jest to be run with --updateSnapshot.

So if you don't use snapshots in your testing, this option shouldn't concern you.

Also note that by default jest is looking for .js, .jsx, .ts and .tsx files inside of __tests__ folders, as well as any files with a suffix of .test or .spec (e.g. Component.test.js or Component.spec.js). You can read about it here.

About the npm ci command

If you are wondering why we are npm ci instead of npm install here is the reason straight from the npm docs.

This command is similar to npm-install, except it’s meant to be used in automated environments such as test platforms, continuous integration, and deployment – or any situation where you want to make sure you’re doing a clean install of your dependencies. It can be significantly faster than a regular npm install by skipping certain user-oriented features. It is also more strict than a regular install, which can help catch errors or inconsistencies caused by the incrementally-installed local environments of most npm users.

Conclusion

That's it. Now every time you push code from your local computer to the origin CircleCi will get notified and it will run your tests.

This was a basic setup of continuous integration and testing.

In the next article, we will add code coverage reports to the process.

Addendum

CircleCI has a cli program that you can install locally on your computer to automate or even run some tasks locally. One of the best features is that you can use it validate your circleci.yml configuration file. Validating the configuration file locally can save you a lot of time, especially when working with more complex workflows.