Since I've started using npm as a JS package manager (maybe back in 2015), I always wanted to publish my own packages which could be either a very particular package I'd use in my side projects or a package that attempts to solve a common problem and help the other devs.
Every time I needed to do that I also wanted to test the workflow of publishing and installing locally. Also, I'd like to see a kind of "preview" of how it'll look like when it's published, the "real" npm page, just to see if the README is ok for example.
After a lot of struggle and attempts with various approaches to solving this problem, I think I finally figure out the best (at least for me) of how to solve those 2 problems.
Before diving deep into the final solution, let me tell you about the problems I had with other solutions.
Symlink
Back in 2016, trying to find a way to do that I saw a lot of people talking about Symlink
.
In short, Symbolic Link
(or Symlink), is when you create a reference link between 2 (or more) files making just a reference to each other.
Imagine you have a library (my-library) and want to use the local files of it in your project (website). A Symlink in this case will be, inside the node_modules
, instead of having the production files of my-library
, it points to the local folder of it.
But... how to do that?
NPM/Yarn Symlink
Of course, I wasn't the only person in the world desiring a proper way of doing that. And that's why both npm
and yarn
provide a way of doing symlink
out of the box.
I won't explain how to use that in this article, but if you still want to know, you can find a link to how to do that.
In a nutshell, what happens is by doing that, npm
or yarn
will:
- Create a global reference to your package;
- Replace the real module with that reference inside your
node_modules
Using this approach solves most of the problem of testing packages locally cross any project... until it doesn't.
Symlink problem
My biggest pain-point with global Symlink was with nested node_modules and how the tools (at least back them) resolves what version of a module A (used in both the project and the library) was supposed to be resolved.
The first time I saw that was while writing a React component library. The workflow was:
- Go to my library
- Run
yarn link
to create a Symlink - Go to my project
- Link
my-library
- Start my dev server
By only doing that, I started having issues with some React internal rules. That was weird because the error message was really true.
After a couple of hours digging into this problem I finally found an issue on React's repo reporting the exact same problem I had and he pointed about about the Symlink:
The answer from the maintainer was clear:
It's not expected that it would work unless you link react back from your module to your app.
This has actually always been the case (React apps are subtly broken when there are two copies of React module). Hooks surface this immediately which I guess is good.
We do have another issue tracking a better error message for this case.
Of course, it makes a lot of sense. In my component library, React was a peerDependency and I didn't ship that within the bundle. Now, using it via Symlink, React was installed in my library AND my project.
Someone posts a workaround for solving that where you would also need to link the react
and react-dom
inside the library and use that link in our project. So my flow would be:
- Go to my library
- Navigate to
node_modules/react
- Run
yarn link
to create a react symlink - Navigate to
node_modules/react-dom
- Run
yarn link
to create a react symlink - Go back to the root level and run
yarn link
to symlink my lib - Go to my project
- Use the link of
my-library
,react
andreact-dom
- Start my dev server
By doing that, my problem was gone! But... gosh. Really?
After finishing my tests I had to remove those 3 links from your project and force install the dependencies.
Doing that a couple of times was ok but after 10 times I got really annoyed and created a bash script to execute those steps for me.
Also, now I'm using Docker to run my projects and I've realized that Symlink does not work with my base setup.
Probably because when I run my container, I only create a volume which the current project folder. When the container is up and tries to use that Symlink, it might need to navigate through my file system and I think that's not possible.
It might be possible to do that by adding some extra configs but I just don't want to. I want an easy way of doing something in my lib, push and install it whenever I need to use WITHOUT polluting my real package at npm.
Also, using Symlink you can't tell for sure if you're shipping all files your application will need to work.
Luckily, I found a very simple way to solve that and I want to share it with you.
NPM Proxy Registry
Companies also want to grasp package management in their projects, but maybe some of them need to be private to protect their business and intellectual property.
NPM offers the service of using private packages but as you can imagine, it charges the company for that.
A solution for that would be using an npm proxy.
An npm proxy is just a server that sits in front of the official npm
registry server and resolves the dependencies to you.
You can publish an internal package using it and instead of the proxy pushing your package to npm
server, it'll store it in its own server.
By running npm install
using a proxy server, under-the-hood you'll pass a list of packages you want to install. If the proxy has a package published in its "database", it'll return to you that package. If not, it'll ask for the NPM server that package and return it to you:
Private packages are one of the capabilities of npm proxies.
Imagine you fork axios and publish to your proxy server that modified version. When you run install, the proxy instead of returning axios from NPM, will return to you the version you've published.
You might have been thinking:
Hmm... If we can have this set up in a cloud server... what if we use that system locally?
Yes... that was the conclusion I've made while observing that solution and that's how we'll tackle the problem of testing packages locally.
There are a couple of options to perform this workflow, but in this article, I'm gonna show you how to do that using Verdaccio.
Verdaccio
Verdaccio is:
A lightweight open-source private npm proxy registry
It's tremendously powerful and in version 4 it has NO config to start using it, which does not necessarily mean we won't need to do configs, but not really in the proxy itself.
Running a local server
The easiest way of using Verdaccio locally is installing as a global npm dependency:
yarn global add verdaccio
## Or with npm
npm install -g verdaccio
After that, you can start the server by running:
verdaccio
If you prefer, you can also run this server with docker:
docker run -it --rm --name verdaccio -p 4873:4873 verdaccio/verdaccio
Keep in mind that when the container will be stopped, the packages you've to publish would be removed.
After running the server, you can check the website at http://localhost:4873
(or http://0.0.0.0:4873
)
Adding your npm user
To be able to publish a package into your local Verdaccio, you first have to register an npm user there. To do that, run:
npm adduser --registry http://localhost:4873 # OR http://0.0.0.0:4873
The information does not need to be secure or accurate. Remember, it's only a local thing! :)
Publishing and consuming
For both publishing and consuming your local package, you have to always specify what's the registry URL. In other words, what's the server npm must find the packages.
One way of doing that is by creating in the root level of the repo you want to consume/publish a file called .npmrc
and specify the registry link there:
# /my-project/.npmrc
registry=http://localhost:4873 # OR http://0.0.0.0:4873
I strongly recommend this method for npm
users. The reason is that npm asks you to sets a global registry via npm config set
or via publishConfig.registry
in your package.json. Both ways are a hassle to rollback when you want to use the regular npmjs.org
registry.
With a custom .npmrc
per project when you wanted to use from the official registry all you have to do is comment out the registry line in that file.
The other option is for Yarn users which consists in specifying the flag --registry
:
# For publishing
yarn publish --registry http://localhost:4873 # OR http://0.0.0.0:4873
# For consuming
yarn add my-private-pkg --registry http://localhost:4873 # OR http://0.0.0.0:4873
By doing that, yarn will resolve the registry without any extra file nor config. If you eventually get annoyed with having to write the registry flag, you can also create a .npmrc
file and yarn will also be able to resolve the registry URL from there.
After this configuration, when you publish or install your local package, npm or yarn will first ask your Verdaccio local server for that package, and Verdaccio will do all the job to store or retrieve local packages and resolve public packages at NPM.
Caveats
When we install a dependency, a bunch of information about it is added in package.lock.json
or yarn.lock
file and one of them is resolved
, the URL where we got that dependency:
// package.lock
{
// ....
"node_modules/my-package": {
"version": "1.6.0",
"resolved": "http://localhost:4873/my-package-1.6.0.tgz",
// ....
}
}
This means that if you commit and push the lock file with the local server URL, wherever your project will be built or tested (like a CI/CD), it will fail because there's no localhost:4887 there.
In that sense, we always want to remember to clean this change before pushing any code.
Also, differently from Symlink where you can just turn a watch
mode to compile the files and see these changes directly in your project, using this approach, you'll need to publish a new version and install that version every new change you've made.
Conclusion
I hope you enjoy this gotcha.
With a simple solution like that, now you're capable to test a production publishing/consuming a package workflow without too much trouble.
Top comments (0)