Gemfury for Javascript Packages

Notice: NPM is the name for 2 disparate tools: a registry that holds javascript packages (npmjs.com), and the CLI to manage javascript packages locally. I talk about both and to distinguish them I use all caps "NPM" for the registry, and all lowercase "npm" for the CLI.

I started a project at work a while back. Specifically, a javascript package for our frontend. It's something that would need to be shared by various projects (I plan to explain the project in a future post). While I could have started the project without worrying about how it would be consumed, I wanted to start thinking about it up front. One of the main known requirements was that it was going to be a private package, at least for now.

The first option was to depend on Github. It’s something that an npm package allows, and wouldn’t be terribly difficult to work with. In fact, it’s something that has authentication built in when a developer has to have access to clone a repo anyways. But we wanted to find a more robust solution. Something specific to holding packages, separate from the tool used to maintain the code.

The obvious answer was NPM. It’s something every javascript developer is familiar with. And it’s easy to use. But, the cost to value ratio is pretty steep. This may not be the case for every team, especially for open source projects. But we decided against it for us for now.

To be honest, I didn’t even know there were alternatives to NPM. When I brought up the question to my team of possible hosting solutions, I learned of at least a handful of options. Some of which were self-hostable, but others which were just general alternatives to NPM.

Then our Infrastructure manager mentioned that we had a Gemfury account. It had a single unused gem in it. We were a ruby shop, so that made sense, but how would that help me for a javascript package? I wasn’t about to wrap my javascript package into a gem just to distribute it. But I took this under advisement.

After some research, I found that Gemfury actually supports a plethora of languages and registry options. While they sound like they would only be useful for ruby gems, they actually work with python's PIP, php's Composer, and even Debian repositories. It’s like a buffet where you can eat some fried rice, gyros, and Texas barbecue all under one roof.

Okay, so they have learned how to scale horizontally in the code registry business, but I care about hosting a private javascript package. How do they compare to NPM?

Gemfury Features

Here are the top features that sold us on Gemfury:

Tokens - Gemfury has this concept of “deploy” (i.e. pull) and ”push” tokens. These are one-way, revokable tokens that you can use to allow people or systems to do a single direction action. For instance, if you only want to allow people to install a package, you can provide them with a “deploy” token. Inversely, if you want to setup continuous deployment to publish changes to Gemfury, you can use a “push” token. These are incredibly powerful for intentionality and security.

There are different ways to use these tokens, but the docs suggest the tokens go into the url. I will show you below how we avoided that in the case of using npm that seems to be acceptable.

Granular User Permissions - With an organization set up, you can add collaborators (we will see how to do that below). These collaborators can have 3 tiers of access to an organization:

• Download Only
• Upload & Download
• Owner (full access) - this one allows someone to add collaborators and manage tokens for an organization.

By adding collaborators you can provide people with access via their own credentials. This prevents you from having to provide a single access token, or manage multiple tokens for multiple users. By doing this, you can add and remove collaborators without affecting any other person or system.

NPM Proxy - This is NPM specific, and I can’t speak for other registry options. You can set up npm to only hit your Gemfury registry to install packages, which may be fine for a very narrow use case. But most projects will need to be able to install from Gemfury first and then move on to NPM if the package can’t be found. Gemfury provides a way to do this (Installing private npm modules).

This actually allows you to name uploaded packages with the same name as one in NPM, but will effectively shadow that package. This can be useful if you want to avoid updating dependencies in many projects, but you have a fork of a public package on NPM that you would prefer to use.

You can also namespace your javascript packages and set up only that namespace to search Gemfury. This is what we have done and what I will demonstrate below. But either way, Gemfury can accommodate.

Mutability - For anyone who has been in web development for a while, you may have been affected by, or at least heard of, the Left-Pad Fiasco. The result of that caused NPM to declare all packages to be immutable. No matter who you are or what you want to do, if you publish something to NPM, it will remain there forever. You cannot delete it, and you cannot overwrite it. (You actually have 72 hours to delete it, and can contact support after that)

This means when you make a mistake, even if no one is being affected by it yet, you have to release a new version. The remnants of that code will remain forever. But, what if you could just rebuild and override the existing version? Or maybe you want to yank it altogether. This may not be the correct course of action, but for better or for worse Gemfury allows it. And I appreciate that. Yield this power wisely.

I will offer some drawbacks here as well, but there aren’t many that seem to affect us:

Token Descriptions - While you can create all the tokens you want, currently Gemfury does not allow you to add descriptions to them. This makes it very difficult to figure out which one is being used where. I’d suggest keeping a secure file to provide descriptions for keys, but hopefully they will add this ability soon. Token Descriptions have been added as of Feb 22, 2019.

Version Tagging - NPM allows you to tag versions as latest, or beta, whereas Gemfury currently does not. They do however support full semver, which allows for suffixes like -beta.0, which provides a well enough work around.

There are some other odd intricacies around authentication methods between using npm and installing gems, but I want to focus on just javascript packages in this post.

Setup

Now we can get into the nitty gritty.

Here are some of the conditions that guided our setup process:

• We have projects that use both npm and yarn CLI tools
• We want it to be easy to revoke access to one user - if they leave or lose their laptop, we don't want to have to inconvenience everyone.
• We want to be able to securely install on CI machines, and similar to users, be able to easily revoke those credentials.

Now here are the steps:

These steps assume you want to create an organization account. Read more about organization accounts here: Gemfury Organization Accounts Documentation. You may not need one for a small group. In that case you should be able to just skip step 1, and the remaining steps will be the same with a single user account, including adding collaborators.

1. Create an organization (Create Organization Link)
2. Have users create their own Gemfury account (Signup Link).
   • They can login via Github, but Gemfury will prompt them to create a new password, which they will need for logging in locally.
3. Add users to the created organization.

These next steps are for local setup:

We use namespacing to make it esaier to configure, but it's not required. It's as easy as appending @organization/ to the package name in the package.json file

1. Add reference to the Gemfury registry in each consuming project's .npmrc file.
   • yarn will automatically pick up the project's .npmrc configuration, but there is a way to only allow yarn to use the configuration via a .yarnrc file.

# ${PROJECT_ROOT}/.npmrc
# Make sure to provide the trailing slash '/'
@namespace:registry=https://npm-proxy.fury.io/organization/

1. Login via npm CLI with Gemfury credentials.

npm login --registry https://npm-proxy.fury.io/organzation/

This will add or edit the users global .npmrc file located in their home directory with this line:

# ~/.npmrc
//npm-proxy.fury.io/organization/:_authToken=${SECRET_TOKEN}

1. Install the dependency via the command line.

npm install --save @namespace/package
# OR
yarn add @namespace/package

With the proper setup, this install step should look in the Gemfury registry first, and then check in NPM if the dependency isn't found.

Now, once a developer is added as a collaborator (step 3) and logs in to Gemfury via the npm CLI (step 5) they will be able to install the dependency with no problem.

Note: We did have some issues wtih yarn versions 1.6 and 1.7, I strongly recommend using the latest versions to avoid any issues.

Continuous Integration (CI) Setup:

As mentioned above, tokens are a great way to offer secure one-way actions. The best use case for these tokens are on your CI system, kept in environment variables. This makes them easy to swap out without affecting any one else, or any other system.

We have 2 different systems that provide different functionality, and thus different setups. Here is how we configured each of them.

For our CI system that handles the package deployment we needed something to build our package and publish it to Gemfury.

1. Create a push token.

1. Add the generated token to CI environment variables. This step wholly depends on your CI. (For reference, I have named mine FURY_PUSH_TOKEN)
2. Upload package to Gemfury. There are a few ways to upload packages to Gemfury. Check out Gemfury Upload docs for more information.

We chose to generate the assets with npm pack and then use curl to push them up to Gemfury.

npm pack
curl -F package=@"$(echo namespace-package-*.tgz)" https://${FURY_PUSH_TOKEN}@push.fury.io/organization/

Note: $(echo namespace-package-*.tgz) just allows us to avoid needing to know the version of the filename, but it does assume there is only 1 tgz file.

For our CI system that handles the consuming application(s) we just need to authenticate before trying to install.

1. Create a deploy token

1. Add the generated token to CI environment variables. This step wholly depends on your CI. (For reference, I have named mine FURY_PULL_TOKEN)
2. In your CI scripts authenticate with npm manually before installing dependencies.

echo "//npm-proxy.fury.io/organization/:_authToken=${FURY_PULL_TOKEN}" > ~/.npmrc
npm install

And that should do it! Feel free to contact me with any questions about our setup. I'd love to help any way I can 🙂

I hope this helped you to get setup with Gemfury. It really is a great service, with an intuitive interface, at a reasonable rate for everyone to get started with.

And while it may not seem intuitive to place javascript packages on a platform originally designed for ruby gems, it really is helpful. It becomes incredibly more powerful when you are with a company that operates in many different languages. Gemfury offers an incredible platform to hold packages of all kinds.

Let me know your experience.