loading...

The chronicles of semantic-release and monorepos

antongolub profile image Anton Golub ・Updated on ・5 min read

Since semantic-release and lerna appeared, the community has made several attempts to combine these technologies together. One does not simply run release in parallel.
Boromir jpg
There are several significant hurdles. The first one is analyzeCommits step. semantic-release tries to resolve a new version by making a cut of commits from the previous release tag in repo, but has no idea how these changes are related to packages.
The second issue is publish. Or prepare. Or rather what happens between.

await plugins.prepare(context);

  if (options.dryRun) {
    logger.warn(`Skip ${nextRelease.gitTag} tag creation in dry-run mode`);
  } else {
    // Create the tag before calling the publish plugins as some require the tag to exists
    await tag(nextRelease.gitTag, nextRelease.gitHead, {cwd, env});
    await addNote({channels: [nextRelease.channel]}, nextRelease.gitHead, {cwd, env});
    await push(options.repositoryUrl, {cwd, env});
    await pushNotes(options.repositoryUrl, {cwd, env});
    logger.success(`Created tag ${nextRelease.gitTag}`);
  }

  const releases = await plugins.publish(context);

When these git add ., git commit -m and git push --tags origin queues from several async "threads" clash in runtime they produce unstashed changes and block each other.
The third problem is cross-dependencies. In most cases monorepository packages are interconnected, so it's necessary to update versions of these references in some way.

2015

atlassian/lerna-semantic-release
If I'm not mistaken, it seems this was the first working solution. The key idea of LSR is to disassemble semantic-release for parts and use them to construct several step-like pipelines and run them with Lerna

# Pre
lerna-semantic-release pre # Set up the versions, tags and commits
# Perform
lerna-semantic-release perform # Publishes to npm
# Post
lerna-semantic-release post # Generates a changelog in each package

Under the hood we see custom commits analyzer, custom changelog generator, custom npm publisher and so on. Unfortunately, this tool design does not provide standard semrel extending features and any other plugins support. Moreover, once semrel changes its inners, this dramatically affects LRS. Still, it was worth a try.

2017

pmowrer/semantic-release-monorepo
Another approach was proposed by Patrick Mowrer. He did not try to overcome the limitations of parallel running, and suggested that releases be performed sequentially.

lerna exec --concurrency 1 -- npx --no-install semantic-release -e semantic-release-monorepo

An important step forward was the use of semrel standard extension techniques. In essence, only one plugin required customization β€” analyzeCommits which was supplemented with commit-by-package filtering. Some implementation flaws are easily fixed with an additional hook, but... cross-dependency problem was not considered at all.

2019

dhoulb/multi-semantic-release
Dave Houlbrooke focused on implantation
of synchronization points inside the release flow. If we cannot should not change the semrel runner code, we can write a plug-in that will provide a coherence state of parallel threads. Each step of synthetic Inline plugin injects wait conditions in order to make sure all concurrent releases are in the same phase.

async function generateNotes(pluginOptions, context) {
    // Set nextRelease for package.
    pkg._nextRelease = context.nextRelease;

    // Wait until all todo packages are ready to generate notes.
    await wait(() => todo().every(p => p.hasOwnProperty("_nextRelease")));

But it’s not enough just to keep parallel releases at one step. Some actions, as we know, lie between plugin[step] calls. And with a moderately large number of packets the problem of unstashed changes appears again. However, it turned out this issue can be easily fixed by replacing internal asynchronous calls with synchronous ones.
The killer feature of this solution is cross-dependencies update. MSR shared milticontext knows about which packages will be updated as part of all releases, so its possible to update all the manifest files too.

// Loop through localDeps to update dependencies/devDependencies/peerDependencies in manifest.
    pkg._localDeps.forEach(d => {
    // Get version of dependency.
    const release = d._nextRelease || d._lastRelease;

    // Cannot establish version.
    if (!release || !release.version)
        throw Error(`Cannot release because dependency ${d.name} has not been released`);

    // Update version of dependency in manifest.
    if (manifest.dependencies.hasOwnProperty(d.name)) manifest.dependencies[d.name] = release.version;
    if (manifest.devDependencies.hasOwnProperty(d.name)) manifest.devDependencies[d.name] = release.version;
    if (manifest.peerDependencies.hasOwnProperty(d.name))
                    manifest.peerDependencies[d.name] = release.version;
            });

2020

qiwi/multi-semantic-release
Dave's solution works great for a small number of packages (<20). But the implemented mechanism that interlocks releases phases is extremely slow for huge "enterprise" monorepos. This fork of dhoulb/multi-semantic-release replaces setImmediate loops and mentioned execa.sync hook with event-driven flow and finally makes possible to run the most release operations in parallel.

// Shared signal bus.
    const ee = new EventEmitter();

    // Announcement of readiness for release.
    todo().forEach((p) => (p._readyForRelease = ee.once(p.name)));

    // Status sync point.
    const waitFor = (prop, filter = identity) => {
        const promise = ee.once(prop);
        if (
            todo()
                .filter(filter)
                .every((p) => p.hasOwnProperty(prop))
        ) {
            ee.emit(prop);
        }
        return promise;
    };
     ...
const publish = async (pluginOptions, context) => {
            pkg._prepared = true;
            const nextPkgToProcess = todo().find((p) => p._nextType && !p._prepared);

            if (nextPkgToProcess) {
                ee.emit(nextPkgToProcess.name);
            }

            // Wait for all packages to be `prepare`d and tagged by `semantic-release`
            await waitFor("_prepared", (p) => p._nextType);

202x

There're a lot of monorepo related issues in semrel repo. NPM's intention to standardize workspace notation brings moderate optimism about the fate of monoreps. As soon as monoreps become more common practice, semrel will probably add built-in support for them.

Discussion

pic
Editor guide
Collapse
wmhilton profile image
William Hilton

Thank you for such a thorough review of the history + state-of-the-art!