Testing npm packages against multiple versions of their peer dependency

If you're maintaining an npm package you know how automated testing takes away much of the pain that comes with it. Especially regression tests help you release with confidence by preventing you from accidentally introducing breaking changes. For most packages, testing the API and running the tests with all supported node versions is enough. Maintaining a plugin, however, comes with further challenges. Typically a plugin supports multiple versions of its host package. An Express middleware, for example, is supposed to work with a range of Express versions. The only way to ensure this is to actually test a plugin against different versions of its host package. I tried to achieve that and with two small tools - package aliases and job matrices - it required only small changes to my existing test setup.

I came across this issue when maintaining the package hapi-auth-any, an authentication strategy for hapi. You don't need to know anything about hapi or hapi-auth-any. I'll just use it as an example of how I adapted my tests after I added support for another major version. I originally wrote the plugin to work with hapi 18. At that point I decided testing against the latest minor version would be sufficient, so my package.json file looked somewhat like that:

{
 ...
 "peerDependencies": {
    "@hapi/hapi": "18.x"
  },
  "devDependencies": {
    "@hapi/hapi": "^18.3.1",
  }
}

By defining hapi 18.x as a peer dependency, I was telling the user of my plugin that they needed to have any version of hapi 18 installed. That wouldn't automatically install hapi when installing the plugin but it would warn them if they didn't meet this requirement. Since peer dependencies aren't automatically installed I also needed to define hapi as a dev dependency so I could import it in my tests like that:

const hapi = require('@hapi/hapi');

Then, I set up a hapi instance, registered the hapi-auth-any authentication strategy and wrote some test cases. All was set.

Some months later, hapi 19 was released. I checked the breaking changes and didn't find any compatibility issues. To ensure that with further development I wouldn't break the compatibility to either version I decided to run the tests against both hapi 18 and hapi 19.

Instead of involving some black pipeline magic I chose a solution that works independent of the pipeline, so any contributor could run the tests against the version of hapi as they like. For this I needed to install two versions of the same package and find a way to tell the test which one it should use. The solution for this problem was package aliases. They were introduced in npm 6.9.0 and allow you to install packages under a different name. That way you can install the same package multiple times and import it using the alias. So all I needed to do was to change the package.json file to look like this:

{
  ...
  "devDependencies": {
    "hapi18": "npm:@hapi/hapi@^18.3.1",
    "hapi19": "npm:@hapi/hapi@^19.0.5",
  },
  "peerDependencies": {
    "@hapi/hapi": "18.x || 19.x"
  }
}

I decided to use an environment variable to tell the tests which version it should use so I replaced the first line of my test file with the following:

const hapi = require(process.env.HAPI_VERSION === '19' ? 'hapi19' : 'hapi18');

If the environment variable HAPI_VERSION is set to 19 it'll use hapi 19, otherwise hapi 18. That makes it easy to use in the pipeline. In order to make it easier for contributors to run the tests I added another test script to the package.json file that uses cross-env to set the environment variable:

{
  ...
  "test": "nyc ava",
  "test:hapi19": "cross-env HAPI_VERSION=19 nyc ava",
}

Now, npm test with no env variable set runs the tests against hapi 18, npm run test:hapi19 runs them against hapi 19.

That was all I needed to do to test against multiple versions of the same peer dependency. Now I only needed to change my GitHub Action so it would test against both versions.

Here, something called a job matrix came to my help. With the help of a job matrix you can generate jobs by substituting specific parts of a job definition. I was already using a job matrix to run my tests with different node versions.

name: Node CI
on: [push]
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [8.x, 10.x, 12.x]
    steps:
    - uses: actions/checkout@v1
    - name: Use Node.js ${{ matrix.node-version }}
      uses: actions/setup-node@v1
      with:
        node-version: ${{ matrix.node-version }}
    - name: install, lint, and test
      run: |
        npm install
        npm run lint
        npm test
      env:
        CI: true

This creates a job for each node version. We can use such a matrix to set the environment variable HAPI_VERSION to 18 or 19 respectively.

name: Node CI
on: [push]
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version:
          - 8.x
          - 10.x
          - 12.x
        hapi-version:
          - 18
          - 19
    env:
      HAPI_VERSION: ${{ matrix.hapi-version }}
    steps:
    - uses: actions/checkout@v1
    - name: Use Node.js ${{ matrix.node-version }}
      uses: actions/setup-node@v1
      with:
        node-version: ${{ matrix.node-version }}
    - name: install, lint, and test
      run: |
        npm install
        npm run lint
        npm test
      env:
        CI: true

Now, the workflow creates a job for each combination of node and hapi version. This seems like what we want but there's a small issue with that: hapi 19 dropped support for both Node 8 and 10. In order to exclude those specific combinations, we can use the exclude key.

name: Node CI
on: [push]
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version:
          - 8.x
          - 10.x
          - 12.x
        hapi-version:
          - 18
          - 19
        exclude:
          - node-version: 8.x
            hapi-version: 19
          - node-version: 10.x
            hapi-version: 19
    env:
      HAPI_VERSION: ${{ matrix.hapi-version }}
    steps:
    - uses: actions/checkout@v1
    - name: Use Node.js ${{ matrix.node-version }}
      uses: actions/setup-node@v1
      with:
        node-version: ${{ matrix.node-version }}
    - name: install, lint, and test
      run: |
        npm install
        npm run lint
        npm test
      env:
        CI: true

That's it. All the tests run now against all the hapi versions and the node versions they support. Even for my small package, there's a benefit in that; for packages with a lot of contributors and bigger releases, this might be even more helpful. With such a setup you can finally release with confidence.