Migrate a 60k LOC TypeScript (NodeJS) repo to ESM and testing become 4x faster (2/2)

In the previous article, we discussed the advantages of using ESM in Node.js and showed the migration steps for a TypeScript project.

But our unit testing was still in CJS (with ts-jest), which is the last piece of our migration and slowed down my M1 Pro MacBook and CI time. In this article, we'll explain how to use native ESM in Jest and provide a minimum ESM repo to cover the key results of this series.

BTW, our project Logto is an open-source solution for auth.

Let's get started!

The disharmony

If you are using Jest, you must have seen this classic error:

SyntaxError: Cannot use import statement outside a module

This usually happens when your Jest runs in CJS mode but meets an ESM module. Searching the message in Google, there are two popular genres (or a combination) to solve this:

1. Introduce a transpiler like babel.
2. For an existing transpiler, fine-tune a complex RegExp to let it work for specific ESM dependencies, e.g.:

// jest.config.js
{
  transformIgnorePatterns: ['node_modules/(?!(.*(nanoid|jose|whatever-esm))/)'],
}

Both of them are OK and just OK. Since our code base is already in ESM, it looks redundant to go back to CJS again.

Challenges

However, everything unusual has reasons. For Jest, they are:

1. As of today (12/26/22), Jest only has experimental support for ESM.
2. ESM is immutable, thus jest.mock() will not work and jest.spyOn() also doesn't work on first-level variables (export const ...). This also applies to other test libraries like Sinon.
3. You may find some libraries for mocking ESM, but almost all of them are creating "a new copy" of the original module, which means if you want to import module A that depends on module B, you must import A AFTER B is mocked to get it to work.

If you are good with 1, just like us, then 2 and 3 will be good, too.

Enable ESM in Jest

Jest has an official doc of ESM as reference. But our situation is slightly different: we are using TypeScript with ESM.

Configuration

Let's temporarily forget ts-jest or ts-node or babel and return to the nature. How about use tsc for transpilation and directly run the JavaScript?

We set up a dedicated tsconfig.test.json for compiling with tests:

{
  // Extends the base config
  "extends": "./tsconfig",
  // Loose some configs for testing
  "compilerOptions": {
    "isolatedModules": false,
    "allowJs": true,
  },
  // Bring back test files we excluded before
  "exclude": []
}

And add some package.json scripts:

{
  "scripts": {
    "build:test": "tsc -p tsconfig.test.json --sourcemap",
    "test:only": "NODE_OPTIONS=--experimental-vm-modules jest",
    "test": "pnpm build:test && pnpm test:only",
    "test:coverage": "pnpm run test --coverage"
  }
}

• build:test is to build all TypeScript files to JS with sourcemap, thus we can have the error stack points to the source code along with the coverage report.
• test:only is to run tests without build, just for convenience.
• test combines build and run.
• test:coverage appends --coverage to generate coverage report. Note some package manager may require an additional -- to pass arguments to the inner command.

For Jest config, remove all presets and special RegExp for supporting ESM or TypeScript, since there's only one key config:

/** @type {import('jest').Config} */
const config = {
  roots: ['./build'], // Point to the build directory
};

To simplify, we changed jest.config.ts to jest.config.js and added @type annotation for type reference in VSCode.

Mocking ESM

At this point, execute pnpm run test should be able to run tests without module system error. If it doesn't, refer to the Jest official doc to ensure config has been replaced correctly.

If you have used the jest namespace functions, like jest.mock() or jest.spyOn(), Jest now complains jest is not defined.

Two solutions:

1. Put a const { jest } = import.meta; before all jest.* calls.
2. Install @jest/globals and use import { jest } from '@jest/globals';.

We chose 1.

But module mock is still not working, which leads to expectation errors in some test suites. Because the API is slightly different under ESM:

jest.unstable_mockModule('node:child_process', () => ({
  execSync: jest.fn(),
}));

Although it has been marked as "unstable", our tests have been running stably for a while without exception.

You may find unstable_mockModule() is not typed for import.meta.jest, so we added the signature manually by module augmentation.

The order matters

By adopting unstable_mockModule(), you can see some tests are fixed, but some are still not. Don't go crazy because we are just one step away.

Because of its immutable nature, ESM CANNOT be edited. This means unstable_mockModule() is creating a new copy of that module instead of update it in place.

Say you have a module bar.ts which imports another module foo.ts (same for JS):

// foo.ts
export const value = 1;

// bar.ts
import { value } from './foo.js';
export const add = () => value + 1;

In bar.test.ts:

import { add } from './bar.js';

const { jest } = import.meta;
jest.unstable_mockModule('./foo.js', () => ({
  value: 2,
}));

describe('bar', () => {
  it('should have value 3', () => {
    expect(add()).toEqual(3); // Error, still 2
  });
});

Because bar loads the original version of foo. Remember ESM has top-level await, so just change to:

const { jest } = import.meta;
jest.unstable_mockModule('./foo.js', () => ({
  value: 2,
}));

const { add } = await import('./bar.js');

describe('bar', () => {
  it('should have value 3', () => {
    expect(add()).toEqual(3); // OK
  });
});

Not hard, right? Actually, the new code makes more sense to us because we find the code becomes predictable. In CJS, jest.mock will be automatically hoisted (that's why we drop var and use let).

You'll also find the following error also disappears:

The module factory of `jest.mock()` is not allowed to reference any out-of-scope variables.

Partial mocking

Due to the same reason, jest.spyOn() doesn't work in top-level variables or functions. E.g.:

// foo.ts
export const run = () => true;

// test.ts
import * as functions from './foo.js';
jest.spyOn(functions, 'run'); // Doesn't work

The essence of .spyOn() a top-level member is partial mocking, i.e., keep everything else the same, but spy on a specific member. So this can be archived by:

const actual = await import('./foo.js');

jest.unstable_mockModule(moduleName, () => ({
  ...actual,
  run: jest.fn(actual.run),
}));

const { run } = await import ('./foo.js'); // `run` has type `jest.Mock`

Looks a little bit verbose, so we made a helper function:

const mockEsmWithActual = async <T>(
  ...[moduleName, factory]: Parameters<typeof jest.unstable_mockModule<T>>
): Promise<T> => {
  const actual = await import(moduleName);

  jest.unstable_mockModule(moduleName, () => ({
    ...actual,
    ...factory(),
  }));

  return import(moduleName);
};

Caveat The moduleName may be problematic if BOTH conditions are met:

• The helper function is not located in the same directory as the caller file.
• The caller is trying to mock a relative path module, e.g. mockEsmWithActual('./foo.js'). Path alias will be fine.

We doubt the reason is importing process is running in a Promise, which leaves the original context. To use relative paths w/o worries, see this file.

Closing note

That's it! Our Node.js code is all in native ESM now, and the dev experience has become much better.

You can find our ts-with-node-esm repo for the key result of this series:

https://github.com/logto-io/ts-with-node-esm

Thank you for reading, feel free to comment if you have any questions!

This series is based on our experience with Logto, an open-source solution for auth.