DEV Community

Cover image for Migrate a 60k LOC TypeScript (NodeJS) repo to ESM and testing become 4x faster (1/2)

Migrate a 60k LOC TypeScript (NodeJS) repo to ESM and testing become 4x faster (1/2)

gao-sun on December 25, 2022

Intro After the third try, we successfully migrated all existing Node.js code from CJS to native ESM, and the CI time of unit testing wa...
Read full post
Collapse
 
webjose profile image
José Pablo Ramírez Vargas •

Yes, I am on board with this too with wj-config. I transpiled it from TypeScript to ES Modules. That's it. Look at some code snippet comparison when used in Common JS and ES Modules:

Set Up

Common JS 

module.exports = (async function () {
    const { default: wjConfig, Environment } = await import('wj-config');
    const env = new Environment(process.env.NODE_ENV);
    return wjConfig()
        .addObject(loadJsonFile('./config.json', true))
        .name('Main')
        .addObject(loadJsonFile(`./config.${env.current.name}.json`))
        .name(env.current.name)
        .addEnvironment(process.env)
        .includeEnvironment(env)
        .createUrlFunctions()
        .build(env.isDevelopment());
})();
Enter fullscreen mode Exit fullscreen mode

ES Modules 

const config = wjConfig()
    .addObject(mainConfig) // Main configuration JSON file.
    .name('Main') // Give data sources a meaningful name for value tracing purposes.
    .addObject(loadJsonFile(`./config.${env.current.name}.json`)) // The developer is deciding by using a file name tactic.
    .name(env.current.name)
    .addEnvironment(process.env) // Adds a data source that reads the environment variables in process.env.
    .includeEnvironment(env) // So the final configuration object has the environment property.
    .createUrlFunctions() // So the final configuration object will contain URL builder functions.
    .build(env.isDevelopment()); // Only trace configuration values in the Development environment.

// This is a top-level await:
export default await config; // The build() function is asynchronous, so await and export the result.
Enter fullscreen mode Exit fullscreen mode

Consumption

Common JS 

const configPromise = require('./config.js');

configPromise.then(config => {
    console.log(config.app.title);
});
Enter fullscreen mode Exit fullscreen mode

ES Modules 

import config from './config.js';

console.log(config.app.title);
Enter fullscreen mode Exit fullscreen mode

Just having top-level await is enough to sway like 90% of people to ES Modules.

Collapse
 
tythos profile image
Brian Kirkpatrick •

We are going through a similar experiment right now, but for migrating to ESM from RequireJS. Ironically, it seems to be much easier, largely because there are nearly zero adaptations/hotfixes required for front-end compatibility, where all the transpilation was destined for anyway. If anything, the weird edge cases come from situations (like automated unit testing) where some degree of NodeJS compatibility is still required.

Collapse
 
romeerez profile image
Roman K •

It was a problem with tooling, not with CJS. Migration of tests from requires to imports shouldn't cause such drastic speed up, it's definitely a bug in the ts-jest or in a related tooling. I'm using @swc/jest for tests and it takes couple of seconds to run a thousand tests written in TS.

The point of this article is that ESM brings a compilation speedup, so what was the change of the build step of your project?

Collapse
 
gaosun profile image
gao-sun •

Thank you! Totally agree there's something that happened in ts-jest or ts-node magnificently slows down the testing speed. Will try @swc/jest.

The faster testing speed is something out of our expectations. The point of this article is to discuss why we decided to transform to native ESM and our know-how of the migration. No big changes for the build steps.

Collapse
 
romeerez profile image
Roman K •

I read more carefully and I see that main reason was a non-compatible packages that enforces ESM to be enabled. That's sad to write extensions in import pathes, have experimental Jest, to loose jest.mock() which I'm using regularly, and all because some libs cannot setup rollup properly to have both commonjs and esm outputs. In your project it's probably important to support all packages, but luckily in other node.js projects it's fine to use an alternative lib.

Thread Thread
 
gaosun profile image
gao-sun • • Edited

Non-compatible packages is a trigger, not the key reason. We kept those PR closed for months.

Extensions and experimental flag are not blockers to us. We don't lose `jest.mock(), as you can see the practice in chapter 2. Actually the code becomes more readable and predictable afterwards.

And it's all about choices and balances - since in most cases our project serves as a service/solution which is not for importing, and our users are using container to run it, so single transpilation strategy is good to us.

We are using rollup for dual-package publish as well, but it's a transitive plan and soon the repo will be pure-ESM.

Collapse
 
konnorrogers profile image
Konnor Rogers •

And now time to swap to Vitest!

Collapse
 
merri profile image
Vesa Piittinen •

Switching to Vitest fixed many issues that we've had with Jest. Like React hooks didn't work as expected within components when running within Jest which had led to cumbersome hook mocking. Especially useContext was very troublesome. I don't know if this is issue always with Jest but the issue did replicate on two different repos that we have.

Vitest is also faster.

Collapse
 
gaosun profile image
gao-sun •

Thanks guys. Will take a look on Vitest!

Collapse
 
micronaut profile image
Eric Danowski • • Edited

We have a code base that I have tried to move to ESM but the problem that I have run into is how to mock imports in a unit test. We use Jasmine and Karma for our specs and if I am testing a module that imports another module (e.g. utils.js), I am unable to stub or mock the exports of utils.js. How are you handling this? You mentioned the issue in the Recap, but I'm wondering if anyone has a solution. Thanks

Collapse
 
gaosun profile image
gao-sun •

Mocking is explained in chapter 2. The principle will be the same.

Collapse
 
micronaut profile image
Eric Danowski •

Thanks!

Collapse
 
jdsalingerjr profile image
Joseph C •

Thank you for the post, this has been really helpful.

But I have a question about the subpath imports. You have the imports defined as:

    "#src/*": "./build/*" // Point to the build directory, not source
Enter fullscreen mode Exit fullscreen mode

Why are you pointing to the build dir?

In your github example you don't have any subpath imports defined. So what is the thought process behind the subpath import pointing to build and further, does it also imply that you need to be running a watcher on the code to continuously update the build output?

Thanks