Transitioning from CommonJS to ESM

This is a translated article from the original post: https://tech.kakao.com/2023/10/19/commonjs-esm-migration/

In this article, we would like to explain why we switched from the old module system CommonJS to ESM and how all this switch happened in the process of upgrading the version of the library used by the service in operation.

• cjs: CommonJS
• ESM: EcmaScript Modules

1. Why we transitioned to ESM

The FE Platform team uses Pharus, a service that periodically analyzes the performance of web services using Google's Lighthouse.

Lighthouse is a static performance analytics tool created by Google that accesses web services in Chrome to collect performance information and generate reports. It is a library that is constantly updated, and versions 10 and 11 were released this year. Version 10 includes major changes such as performance score changes, Typecript support, and transition to ESM.

Pharus, on the other hand, uses Lighthouse to measure the performance of a web service every six hours and makes it easier for developers to keep track of changes in performance. It also supports the authentication processing needed to access a web service to measure performance, as well as setting up user agents and headers.

Pharus is constantly following Lighthouse's version updates to support performance measurements in the latest version of Chrome, and our goal was to apply the new version of Lighthouse (v.10) in Pharus.

Previously, Pharus was loading Lighthouse v.9 in the CommonJS environment. Because the module system switched from CommonJS to ESM, the require() function of CommonJS does not allow the module in the ESM system to be loaded. In response, Lighthouse v.10 provides two ways
for CommonJS environment compatibility.

1. await import(‘lighthouse’)
2. require(‘lighthouse/core/index.cjs’)

However, if we were to apply both methods, many changes had to be made, and also there were restrictions on using some features, so we switched to ESM so that Pharus (which is based on Lighthouse v.10) could load the modules.

(1) Importing through dynamic import

Although option no. 1, await import(‘lighthouse’), was not selected, we'll explain the process we took.

Pharus is written in Typescript and compiled into module syntax that can be interpreted in a Common JS environment by setting tsconfig.json as follows.

// tsconfig.json
{
  "compilerOption": {
    "module": "CommonJS"
  }
}

Then, we changed the code to get Lighthouse using await import() as below.

// import lighthouse from ‘lighthouse’;
const lighthouse = await import('lighthouse');

However, an error occurred that the require() could not get the ES Module.

Error [ERR_REQUIRE_ESM]: require() of ES Module ... not supported.

This error occurs because when module property is set to "CommonJS" in tsconfig, await import() is converted to "require()". Therefore, we changed to "Node16" again as below.

// tsconfig.json
{
  "compilerOption": {
    "module": "Node16"
  }
}

If we try this again, we will get an error saying that the await syntax should be used within the async function.

const lighthouse = await import('lighthouse');
                   ^^^^^

SyntaxError: await is only valid in async functions and the top level bodies of modules

Although the await import() allows you to import aa ESM Lighthouse module in a CommonJS environment, the await syntax can only be used within the async function, so more code need to be added to import the module. In the end, the await import() was discarded.

(2) Importing as cjs file

Option no.2, require(‘lighthouse/core/index.cjs’), was not selected due to some functional constraints. Here's what happened.

Lighthouse still provides index.cjs files for existing CommonJS users. index.cjs imports some modules from index.js, which is the entry point for the Lighthouse, into await import() and then exports them using module.exports.

Because Pharus defines and uses a new measurement environment by expanding the measurement elements in Lighthouse, we needed to import and use Gatherer and Audit classes. However, since index.cjs does not export these classes, importing index.cjs alone could not solve the problem.

As such, we decided to switch Pharus to ESM because the functionality of the Lighthouse was not available in the CommonJS environment.

2. Differences between CommonJS and ESM

Common JS and ESM have different syntax for exporting and importing modules. For now, they are not compatible with each other, so you should use the appropriate syntax.

Now, let's get to the point where Node.js decides which module system to use to parse files, and see the difference in syntax between CommonJS and ESM.

Determining the module type

This is the way Node.js decides how to parse JavaScript files. Each order means priority, and the order determines the module system if the conditions are correct. More details are available in the Determining Module System document.

1. Check the file extension. The .cjs extension is CommonJS, whereas .mjs is for ESM.
2. Check the 'type' field in the closest upper package.json. If its value is 'module', then it's ESM, whereas if it's 'commonJS' or empty, then it's CommonJS.
3. Check Node.js --input-type flag. --input-type=module is ESM, while --input-type=commonjs means it's a CommonJS.

Exporting the modules

CommonJS

• module.exports

module.exports = 'module';

module.exports = {
  someNumber: 123,
  someMethod: () => {}
}

• exports: When you export multiple values from a single file, you can pass them to the properties of the exports rather than as objects in module.exports. exports and module.exports have the same reference, so if a different value is assigned to exports the reference to module.exports disappears and cannot be exported.

exports.someNumber = 123;
exports.someMethod = () => {};

// although this will not be evaluated, it will still be exported
if (false) {
  exports.falseModule = 'false';
}


// modifying exports will prevent the module to be exported
(function (e) {
  e.aliasModule = 'alias';
})(exports)

// not exported
exports = 'abc';

// not exported
exports = {
  something: 123
}

ESM

• export default: You can export one particular module value to the default export.

export default {
  something: 123
}

• export: You can export values to the specified name.

export const namedVar = 'module';

• export from: Imports a module and exports it right away.

export otherModule from './otherModule.js';

Importing modules

CommonJS

• require(): We use the require() function when importing modules in CommonJS, which cannot import ESM files (ERR_REQUIRYRE_ESM error occurs when trying to import ESM files). Also, you do not need to create a file extension.

const module = require('./moduleFile');

• import(): Imports ESM modules asynchronously in CommonJS. You must specify a file extension.

import('./moduleFile.js').then(module => {
  ...
});

ESM

• import: Use import statements to import ES modules. Dynamic values are not evaluated, and therefore not available, because the module is imported in the parsing phase. Both CommonJS and ESM modules can be imported with this syntax and you must specify a file extension.

import { functionName } from './moduleFile.js';
// imports all the named exports 
import * as allProperty from './moduleObject.js';
// Don't
import {AorB_Module} from condition ? './A_module.js' : './B_module.js';

• import(): Dynamically imports modules.

const module = await import('./moduleFile.js');

const aOrb_Module = await import(condition ? './A_module.js' : './B_module.js');

Using the CommonJS Modules in an ESM Environment

Still a lot of libraries in Pharus use CommonJS. Now let's look at how to import CommonJS modules when you switch to ESM. First, you can import CommonJS modules as import statements or import expressions in ESM, unlike CommonJS' require().

// Modules exported as module.exports are exported in default properties.
import { default as cjsModule } from 'cjs';

// For convenience, you can import it without using { default as cjs} the way ESM uses it.
import cjsSugar from 'cjs';

// The cjsModule and cjsSugar have the same value.
// <module.exports>

import * as module from 'cjs';
const m = await import('cjs');

// The module and m have the same value.
// [Module] { default: <module.exports> }
// module.default has the same value as cjsModule.

CommonJS may also use exports to export multiple modules by giving a specific name in a file. This is because cjs-module-lexer is integrated into Node.js, which allows us to import modules of CommonJS individually in an ESM environment. This method can work independently of the evaluation of the value because it imports modules from the parsing stage.

Because the syntax is different, import/export statements are not available in CommonJS and require/module.exports is not available in ESM. We need to export and import each module using the appropriate syntax.

3. Set to ESM environment

Let's set the service environment to the ESM system.

Module type configuration

To transform the environment of the service into an ESM system, change the extension of all files to mjs or change the type property from package.json to "module" as shown below.

// package.json
{
  "type": "module"
}

TypeScript configuration

When written in Typescript, the module syntax gets converted into a specific one. It supports the module property to determine which module system to compile the module import/export syntax into. And by default it's "CommonJS". Set compilation options so that when Typescript gets compiled, it can be built into ESM's import and export statements.

It gets compiled into ESM syntax if we use ES2015, ES6, ES2020, ES2022, ESNext values.

// tsconfig.json
{
  "compileOption": {
    "module": "ES2020"
  }
}

4. Learn how ESM works under the hood

We experienced two issues in transitioning Pharus from a Common JS environment to an ESM. But before we have a look at the solutions to the issues, let's first learn how ESM works. More information is available here.

ESM operates in three stages: configuration, instantiation, and evaluation. After getting module resources(files), parsing, setting the memory address for each variable or function, and finally executing the code and populating the values.

source: https://hacks.mozilla.org/2018/03/es-modules-a-cartoon-deep-dive/

Here's what happens in detail:

• Configuration: Gets module resources and performs parsing.
• Instantiation: Allocates and shares the memory address of the variable that each module imports and exports.
• Evaluation: Fills a value into memory while executing the code.

Unlike CommonJS, which performs each step synchronously while traversing all modules, ESM performs each step asynchronously, allowing one module to use the await syntax at the module top level, such as the async function, and consequently improving performance for the entire file load.

Let's dig into each one them in more detail.

(1) Configuration

Source: https://hacks.mozilla.org/2018/03/es-modules-a-cartoon-deep-dive/

1. Check where to import the file containing the module. After parsing, the **Loader checks the module designator in the import statement and determines the file path according to the module resolution rules (initially checks the entry point file).

2. Imports the file.
   Before importing them, it checks the module map to prevent duplicate files from being imported. Then import process is initialized. Different loaders can be used depending on the platform. For example, browsers import files through HTTP communication based on HTML specifications, while Node.js imports them using the file system. Node.js provides the ability to customize the Loader.

3. Parse the file into a module record.
   The file itself cannot be read by the browser or Node.js. Therefore, it gets parsed into a module record that contains an Abstract Syntax Tree (AST).
   When parsing is complete, the process is repeated until all linked files are imported.

The tree-structure module record is created once the repetition is over.

(2) Instantiation

The detailed process of ESM's instantiation process is as follows.

Source: https://hacks.mozilla.org/2018/03/es-modules-a-cartoon-deep-dive/

The Javascript engine configures the module environment records per module record. The module environment records manage the variables in module records and allocate and track the memory of each variable. Depth-first search(DFS) is applied to the module records tree to assign memory addresses to module records from modules that only export without dependencies on other modules.

Source: https://hacks.mozilla.org/2018/03/es-modules-a-cartoon-deep-dive/

At this time, the module that is importing will share what memory address the imported variable has.

CommonJS replicates each module value, but ESM shares the memory address

Source: https://hacks.mozilla.org/2018/03/es-modules-a-cartoon-deep-dive/

Since ESM shares the memory, we cannot assign a value to an imported module on an imported file. Otherwise we might end up having side effects.

At the end of the instantiation process, the memory locations for all variables or functions that have been imported/exported get associated.

(3) Instantiation

To fill the actual memory with values, the JS engine executes the code from the top. To avoid possible side effects, the module gets evaluated only once. For example, if a module calls a server, the results may vary depending on the number of evaluations. Through the evaluation process, a module map is constructed so that each module has one module record, allowing each module to be evaluated only once.

To be continued in part II...