Tim Nelles

Posted on Nov 21

Syncing an NPM Package Between Multiple Registries

#npm #gitlab #cicd #javascript

The Problem

Our team relies on an internal GitLab NPM registry for managing packages. This works perfectly for internal distribution, but there are instances where a package needs to be shared with external collaborators. Syncing the package from the internal registry to an external one became a new challenge.

I set out to find a solution and, as usual, encountered a series of hurdles that turned this into another chapter in my developer struggles.

Attempt 1: Using HTTP Requests to Mimic NPM CLI

Initially, I thought I could handle this with raw HTTP requests, essentially mimicking what the NPM CLI does under the hood. The idea was simple: query the source registry for package metadata, fetch the tarball, and upload it to the target registry.

Why it failed:

• The process was too complex and fragile.
• It required manually constructing the entire package.json manifest for every version.
• Every small misstep led to broken packages.

Attempt 2: Leveraging npm-registry-sync

Next, I discovered npm-registry-sync, a library designed for syncing NPM registries. This tool almost solved the problem; it could monitor changes and replicate them across registries.

Why it didn’t work for me:

• It operates in “daemon mode,” polling for updates continuously.
• In a GitLab CI pipeline, I needed a one-off execution, controlled entirely by the pipeline — no background processes allowed.

The Winning Solution: The NPM CLI

Eventually, I realized I could stick to the tried-and-true NPM CLI. The steps were straightforward:

Install the package locally from Registry A.
1. Reconfigure NPM to point to Registry B.
2. Publish the package to Registry B.

While this worked like a charm, there were a few extra steps needed to make it CI-friendly.

Configuring NPM for Multiple Registries

Managing registry configurations dynamically in a CI pipeline was a bit tricky.

Here’s how I solved it:

General Configuration

Using the npm CLI, you can set parameters for each registry:

npm config set "//my.awesome.registry.com:<parameter name>=<parameter value>"

Important Gotcha:
The URL in the config must exclude the protocol (https:).

Package-to-Registry Association

To associate a specific namespace or package with a registry:

npm config set "<your namespace>:registry" "<your registry url with https:>"

Handling Authentication in CI Pipelines

Some registries required a username/password combo, while others used tokens. Here’s what I learned:

Tokens

Tokens are straightforward, but ensure you strip the protocol when configuring the auth URL:

npm config set "//my.registry.com:_authToken=<token>"

Basic Auth

Generating a basic auth hash (username:password) required attention to detail. In some distros, the base64 command has quirks that differ from others.

On macOS:

echo -n "<my username & password hash>" | base64

Will give you (as expected):

d2hhdCBhcmUgeW91IGRvaW5nIGhlcmU/IGdvdCB5YSEgc29tZSBtb3JlIHRleHQgdG8gbWFrZSB0aGlzIHJlYWxseSByZWFsbHkgbG9uZw==

With docker linuxkit:

d2hhdCBhcmUgeW91IGRvaW5nIGhlcmU/IGdvdCB5YSEgc29tZSBtb3JlIHRleHQgdG8gbWFrZSB0
aGlzIHJlYWxseSByZWFsbHkgbG9uZw==

What is going on here? There is a line break!

On some distros, the wrap parameter has a default set to 76 chars for formatting private keys etc.

It works like this:

echo -n "<my username & password hash>" | base64 --wrap 0

The echo -n is very important. If omitted, echo will add a linebreak to the end of the string and this will manipulate your hash.

The finished script

Putting it all together, the script could look like this:

#!/usr/bin/env bash

# Input validation
if [ "$#" -ne 5 ]; then
  echo "Usage: $0 <source-registry> <target-registry> <source-registry-token> <target-registry-username> <target-registry-password>"
  exit 1
fi

SOURCE_REGISTRY=$1
TARGET_REGISTRY=$2
SOURCE_REGISTRY_TOKEN=$3
TARGET_REGISTRY_USERNAME=$4
TARGET_REGISTRY_PASSWORD=$5

echo "SOURCE_REGISTRY=$SOURCE_REGISTRY"
echo "TARGET_REGISTRY=$TARGET_REGISTRY"
echo "SOURCE_REGISTRY_TOKEN=${SOURCE_REGISTRY_TOKEN:0:10}"
echo "TARGET_REGISTRY_USERNAME=${TARGET_REGISTRY_USERNAME}"

echo "Syncing from '$SOURCE_REGISTRY' to '$TARGET_REGISTRY'..."

remove_protocol() {
  sed -E 's/^https?://g;t' <<< "$1"
}

packages=(
  "@my-namespace/my-package"
)

# INFO: Removing the protocol (^https:) from the authToken config is crucial. Will not work without it.

# Auth for source registry
npm config set "$(remove_protocol $SOURCE_REGISTRY):_authToken=$SOURCE_REGISTRY_TOKEN"

# --wrap 0 is super important! If omitted, it will cause the base64 to contain a line break after 76 chars!
BASIC_AUTH=$(echo -n "$TARGET_REGISTRY_USERNAME:$TARGET_REGISTRY_PASSWORD" | base64 --wrap 0)

# Auth for target registry
npm config set "$(remove_protocol $TARGET_REGISTRY):_auth" "$BASIC_AUTH"
npm config set "$(remove_protocol $TARGET_REGISTRY):always-auth=true"

# Set to source registry to fetch metadata
npm config set "@my-namespace:registry" "$SOURCE_REGISTRY"

for package in "${packages[@]}"; do
  echo "Syncing '$package'..."

  for version in $(npm view "$package" --json | jq -r '.versions[]'); do
    # Install locally
    npm install "$package@${version}" --ignore-scripts

    # Switch to target registry
    npm config set "@my-namespace:registry" "$TARGET_REGISTRY"

    # Publish on target
    npm publish ./node_modules/"$package"

    # Re-set to source registry
    npm config set "@my-namespace:registry" "$SOURCE_REGISTRY"
  done
done

echo "Done syncing packages."

So you could use it like this:

#!/usr/bin/env bash

SOURCE_REGISTRY=https://my.source.registry.com
SOURCE_REGISTRY_TOKEN=abc-xyz-1234568
TARGET_REGISTRY=https://my.target.registry.com
TARGET_REGISTRY_USERNAME=john_doe
TARGET_REGISTRY_PASSWORD=supersecret123

sync-registries.sh $SOURCE_REGISTRY $TARGET_REGISTRY $SOURCE_REGISTRY_TOKEN $TARGET_REGISTRY_USERNAME $TARGET_REGISTRY_PASSWORD

Key Lessons Learned

Stick to Simple Tools:
The npm CLI might not be fancy for this task, but it’s reliable and gets the job done.
Mind the Details:
Configuring authentication, especially with base64, can have subtle platform-specific quirks.
Keep It CI-Friendly:
Avoid solutions like daemons or background tasks when working in CI/CD pipelines. Keep the process under pipeline control.

Syncing npm packages between registries was a frustrating but rewarding learning experience. If you’re facing a similar challenge, I hope these lessons help you navigate the struggle with a bit more ease!

DEV Community