DEV Community 👩‍💻👨‍💻

Cover image for What is pnpm? Is it really so fast and space-efficient?
Sylwia Vargas for StackBlitz

Posted on

What is pnpm? Is it really so fast and space-efficient?

We announced at ViteConf that our WebContainers now support pnpm. It was a major achievement in our commitment to support the Vite ecosystem as many projects running on Vite also use pnpm (examples include Vue, Astro, SvelteKit).

In this post, I’d like to talk about what sets pnpm apart from other package managers - and why we need package managers at all. As a person who best operates through analogy, I will also talk about: 🍣 sushi, 🪦 cemetery, 🎁 birthdays, and 🖼 memes.

What are package managers?

I've also written an “Explain like I’m 5” version of this section!

Let’s start with the terminology.

A surrealist image of massive clouds with a shiba-inu dog head that is consuming a city. The clouds are labelled "node modules" and the city is labelled "github"

package, dependency, node module

A “package” (or “dependency”) is one or a few files bundled neatly together that can be downloaded from a package registry. Once it’s downloaded and installed, it resides in the node_modules folder in a project’s directory. In this post, I will use the words a Node module, a package, and a dependency interchangeably as many folks in the tech community do. That being said, the npm docs offer a differing opinion so check it out if you’re interested in trivia and precision. 🙂

package.json

Each package has a file called package.json which contains information on the package author, its version, its dependencies, and so on. A package can have many dependencies, which in turn can also have many dependencies (called "sub-dependencies"). This sounds like a lot of files, doesn’t it! It really is, especially if your project requires bigger packages.

In the olden times, you’d manually download, install, and configure the dependencies of your app. Keep in mind that most packages are frequently updated, which means that you’d have to always make sure that your project's dependencies (and their sub-dependencies!) are up to date - or that in a given project you use a correct version of a given package. This sounds not only stressful but also unnecessarily repetitive! This is where a package manager comes in.

package managers

A package manager is a tool that helps you keep track of all dependencies of your app in a consistent way. It automates the tasks of dependency installation, upgrading, configuration, and removal. It makes sure that all the packages that your app needs are installed, of the correct version, and up-to-date. This frees so much time for coding… and procrastination 🤪

The first package manager for the Node runtime was introduced only in 2010 and it was npm, (which, contrary to popular belief, doesn’t stand for “Node package manager” and, in fact, is an empty acronym). Twelve years later, npm is now managed by Microsoft and there are two other contenders to the iron throne of the package universe: yarn and pnpm.

What do package managers do?

A package manager grabs the published source files from the online registry and installs them into a directory (a folder) called node_modules. This is the folder where your app will look for the packages when you call the require() method.

You can think about the node_modules folder as a box with helpful tools that you can plug into your app so it works in a desired way. Except that all the tools are made of iron (or osmium!) and the box is massive. What I mean to say is that, sadly, prior to pnpm the node_modules folder would usually be bulky and take a lot of disk space. This resulted in a collection of memes such as:

A popular meme portraying heaviest objects in the universe such as Sun, neutron star, black hole, and Node modules

Imagine that you have numerous projects on your local machine, and each project comes with its own node_modules folder. Most likely, many of these projects use a similar stack. In such a case, you end up with the same packages in numerous projects. Duplication and redundancy! Think about all those inspired side projects that never left the boilerplate stage, all the past attempts to check out all the hot new frameworks - all of them are chilling on your precious disk space.

A comic strip where a computer folder containing Windows brags about being the biggest just to be intimidated by the "Node modules" folder

Who cursed the node_modules folder?

To understand these jokes, we need to take a stroll down memory lane and look at how package management has traditionally been handled.

To the best of its intentions and abilities, npm handled the dependencies by splitting the installation (which is triggered by, for example, npm install ) process into three phases:

  1. Resolving, when the package manager is checking all the project’s dependencies (and their sub-dependencies) listed in the package.json file, finds a version that satisfies the version specifier (for instance, ^1.0.0 would install any future minor/patch versions). Afterwards, it creates a file like package-lock.json for npm and pnpm-lock.yaml for pnpm. you can think about it like a project’s equivalent of a birthday gift wishlist.
  2. Fetch, when the package manager takes the list of resolved dependencies and fetches all the packages from the package registry, which is an equivalent of the feverish shopping in a crowded mall two hours before the party for the gifts all of your friends will give to the birthday person.
  3. Linking, when the package manager writes all the dependencies into the project’s node_modules folder, which, finally, is an equivalent of placing all the gifts in the corner of the party room for when they are needed.

In this scenario, each phase needs to end for the next one to begin. This means that if one dependency has twenty dependencies itself or if one package takes forever to be downloaded, you may have to wait for a long time. If we follow the birthday analogy, imagine that the wishlist included a new version of a popular phone and a box of chocolates. You end up queuing for a few days in front of the phone store to only then buy the chocolates. In such a system, you can’t “save” your spot in the queue, go buy the chocolates and join the party, even if that looked like a better use of your time. You see that t*he way npm manages its tasks is just not efficient.*

As we said earlier, it is also the matter of the disk space. You will end up with numerous duplications of the same packages. And even if there are different versions of the same package, it is rarely the case that all of the files have been changed from one version to another. You still end up with some number of copies. We can agree that the way npm manages the disc space is not efficient either.

How is pnpm a solution to all our worries and troubles?

On its docs page you can read that pnpm is a “fast, disk space efficient package manager.” It really is fast - locally, up to three times faster than the alternatives - and space-efficient. But - how?

pnpm is fast

Do you remember the birthday wishlist analogy? As a gofer, npm first collected all the wishes, then bought all the gifts, then placed them in the corner of the party room. Each phase needed to end for another to begin. This is how the npm gofer organizes its way through life.

Well, if pnpm were the gofer, they’d buy the gift as soon as they’ve read the wish, and designated the spot in the room as soon as they placed the order. They may not even have the physical gift yet to already be able to mark the space in the room where the gift will stand. This happens for each gift separately and independently from others. By design, pnpm doesn’t have the blocking stages of installation - the processes run for each of the packages independently.

pnpm is disc space efficient

Let’s talk about food. I love veggie sushi. I order a lot of veggie sushi. I also eat a lot of wasabi and I don’t like waste. Even though it would be fair to assume that such portions surely are to be shared, I usually order just for myself. My order always arrives with two pairs of chopsticks even though I say on the phone that I don’t need them - I have metal ones at home. I wouldn’t just throw unused chopsticks away and now I have a drawer full of them, and you can also see them laying around in the most surprising places in the kitchen. Similarly, I have my own wasabi tube but would I throw away those little sachets? Never. They chill on the shelf in my fridge in an ever-expanding fashion.

My chopsticks and wasabi could work as an analogy for how npm manages disc space. “Oh, you’ve already installed React two hundred times? I’m SURE you NEED the 201st copy!”

To my relief, pnpm checks what is already available on your disk and only adds what you additionally need for your project to run. All the dependencies are located in one global location (for instance, ~/.pnpm-store/ - you can check it by running pnpm store path in the terminal) called “a content-addressable store”. In your project’s node_modules folder there is a .pnpm file that contains the “virtual store” with many so-called “hard links”. it creates one hard link for each file of each package. I like this explanation that pnpm docs offer:

So, for example, if you have foo in your project as a dependency and it occupies 1MB of space, then it will look like it occupies 1MB of space in the project's node_modules folder and the same amount of space in the global store. However, that 1MB is the same space on the disk addressed from two different locations. So in total foo occupies 1MB, not 2MB.

This makes the node_modules folder more like a portal (like the wardrobe in Narnia) to files located in different nooks of the global store, and not like a bloated storage unit. This diagram illustrates the strategy employed by pnpm:

A graph explaining how pnpm store works

But, does pnpm “know” if there are duplicate files between two versions of the same package? Yes, because, just like git, it identifies the files by a hash id (called also “content integrity” or “checksum”) and not by the filename. This means that two same files will have identical hash id and pnpm will determine that there’s no reason for duplication.

pnpm has got your back

There are so many ways that pnpm looks out for you. One of them is that it is impossible to invite silly bugs by trying to use modules that are not directly specified in the project's package.json but, for instance, are required by your project’s dependencies. While it’s not the end of the world if everything works well, what happens if your project’s dependency no longer requires one of the packages and as a result it is no longer available in the the node_modules folder? Well, everything crashes and you don’t even know why.

Such bugs may happen in npm and Yarn Classic because of the flat node_modules directory because during installation, they hoist (elevate) all of the packages to the node_modules , regardless of whether they are directly required by your app or not. As a result, your project gets access to distant “relatives” of its dependencies.

…but what if my project doesn’t support symbolic links?

And even if your project presents an edge case and doesn’t work well with symbolic linking, worry not! You can still use pnpm but set it to an npm-like mode. While it will not be space-efficient, it will still be faster.

But… what about Yarn?

There are two editions of Yarn:

  • “Yarn Classic”, which comprises versions below v2 and is no longer maintained,
  • Yarn Berry”, which is v2 and higher.

Yarn Classic operates similarly to npm with regards to managing the node_modules folder. Yarn Berry, on the other hand, offers three solutions:

  1. the npm-like mode
  2. the “Plug'n'Play” mode
  3. the pnpm-like mode, which uses hard links to reduce the disc space (not available by default and not thoroughly documented)

The Plug’n’Play seems like an intriguing option but it has been met with the community’s apprehension as the .zip files are not so easily accessible during the dev workflow. The earlier-mentioned post by TakeShape explains some of the other challenges.

Closing remarks

Considering how space-efficient and fast pnpm is, it is not a surprise that more and more projects make a move towards it and that its popularity is rapidly growing. This year, the weekly download rate for pnpm was seven times as high as last year!

If you’re interested in learning about why projects choose to switch to pnpm, check out these posts by Gestalt and TakeShape.

Further reading

Here are further readings that may give you a better understanding of the current landscape of package managers:


Come work with us on package managers!

While you’re here, an announcement: we are looking for a new team member who will work on package managers including pnpm, npm, yarn, and our own Turbo. Interested? Apply today or reach out to me on Twitter!

Top comments (19)

Collapse
 
derlin profile image
Lucy Linder

Very interesting, thanks!
How does pnpm manages cleanup though? With npm, you delete the project folder, and node_modules disappear. Is pnpm able to detect some deps in the cache are now dangling and useless?
If not, does it mean cleanup requires deleting the pnpm folder, and re-run pnpm install on all projects? (which no one will do and thus the pnpm folder may grow indefinitely?)

Collapse
 
sylwiavargas profile image
Sylwia Vargas

Thank you for your questions, @derlin!

Yes, the pnpm cache grows indefinitely basically but there is usually a lot of overlap of dependencies between projects. Pruning the store every once in a while is a good idea. You can do it via pnpm store prune, which removes unreferenced packages that are not used by any project.

Collapse
 
ndaidong profile image
Dong Nguyen

great question, I use pnpm daily and I see it has pnpm prune command, but I never tested it. If it works as same as docker volume prune, it's exactly what we need.

Unfortunately, even the document:

Removes unnecessary packages

It doesn't seems easy to understand how it actually works.

Image description

May Sylwia help us to clarify?

Collapse
 
sylwiavargas profile image
Sylwia Vargas

Thank you, @ndaidong! I think we posted at the same time - yes, you're right about pnpm prune!

Thank you for your questions, @derlin!

Yes, the pnpm cache grows indefinitely basically but there is usually a lot of overlap of dependencies between projects. Pruning the store every once in a while is a good idea. You can do it via pnpm store prune, which removes unreferenced packages that are not used by any project.

Thread Thread
 
ndaidong profile image
Dong Nguyen

You're rock! We need pnpm store prune. Just cleaned :)

Image description

Collapse
 
madza profile image
Madza

Insightful and a great alternative 👍✨

Collapse
 
sylwiavargas profile image
Sylwia Vargas

Thank you!

Collapse
 
fyapy profile image
aabdullin

PNPM not secure

Collapse
 
thi3rry profile image
Thierry Poinot

Could you say more about that ?

Collapse
 
fyapy profile image
aabdullin • Edited on

IT companies of Belarus and Russia encountered problems in the work, to be exact one of the authors has sewn the code that does not allow to run CLI from the territory of these countries, such childishness neither the authors of NPM, nor the authors of Yarn do not do.

So the use of PNPM is a potential problem for any of the countries if the author of the bibliography, you regular business didn't like something, lol.

Thread Thread
 
sylwiavargas profile image
Sylwia Vargas

Would you provide links where this is documented?
I've set my VPN to Belarus and I'm able to run it.

Thread Thread
 
fyapy profile image
aabdullin • Edited on

This is PNPM website from Belarus IP address

Image description

Thread Thread
 
thi3rry profile image
Thierry Poinot

Ok I see, I think he is mentioning the pnpm decision on twitter : https://twitter.com/pnpmjs/status/1498306992577957890?s=46&t=0bwOqnztoi2cUIkmGvGBow

Thread Thread
 
sylwiavargas profile image
Sylwia Vargas • Edited on

Yes, I see that on my end as well. Given that the author of pnpm is from Ukraine and still in Ukraine, I don’t find this surprising.

(EDIT: I see that he was also open about this decision so nothing sneaky there)

However, you were talking about CLI - could you provide links?

Blocking a website based on the location is not a security threat, especially if the website and docs are open sourced and accessible on GitHub.

Thread Thread
 
fyapy profile image
aabdullin

What I mean is that CLI commands just don't work from specific IP addresses, that is, if your servers are located in 4 countries, there is a risk for your business that your batch manager will stop working at some point because of a free decision of the library's author, which in serious projects is not acceptable.

If you want to reproduce the problem, the ideal solution would be a server in Belarus. Try to install PNPM, and install some package.

Thread Thread
 
sylwiavargas profile image
Sylwia Vargas • Edited on

I hear you and what you’re saying is not overlapping with the experience of my friend in Belarus nor with my experience on VPN. Would you provide links to where this issue is documented?

You also mentioned this as “childishness” and “serious project” but this is a common practice in tech - whether it’s good or bad, that’s a subject for opening a discussion. To give you an example, GitHub blocked Devs from Iran, Syria, and Crimea two years ago and here’s a whole list of serious business blocking Iran. Here’s a Wikipedia entry on GitHub’s track record in this field. It is a common practice by the protect authors or whole businesses.

A discussion about political decisions of a project or a business and its merits is one thing. You’re throwing an accusation without documentation. I’m not saying that what you’re saying is not true but so far I haven’t managed to see evidence of that and you’ve avoided providing me with one, even if it’s a link to an issue or a tweet which engaged Zoltan. Moreover, Zoltan was asked if the CLI will be affected:

to which he responded:

I understand that a decision like this is bound to trigger responses and emotions - and it’s fair to express them.

Thread Thread
 
zkochan profile image
Zoltan Kochan • Edited on

As Sylwia already stated, only the website with the docs is blocked. The CLI works. Also, the standalone install script doesn't work because it is a script from the website. Other install methods work, like corepack enable and npm i -g pnpm.

However, if companies in Russia and Belarus decide not to use pnpm, my goal is achieved. I don't want my work to help such companies and people. I live in Ukraine, my life is in constant danger because of Russia and Belarus.

Collapse
 
dvdvdmt profile image
Dmitriy Davydov • Edited on

So, if you don't need the 3x speed increase for the dependencies installation and don't want to reduce some disc space consumption then you can continue to be happy with npm 🤷‍♂️
There is a feature parity between pnpm and npm, but the later wins because it is shipped with Node and it has much broader adoption. I think that all the innovation from the pnpm eventually will come to the npm, it is just a matter of time.

Collapse
 
sylwiavargas profile image
Sylwia Vargas

you can continue to be happy with npm

Yes! It's always good continue to be happy with whatever makes you happy 🙂
I appreciate your optimism about npm roadmap - let's hope it will be that way.

DEV

Thank you.

 
Thanks for visiting DEV, we’ve worked really hard to cultivate this great community and would love to have you join us. If you’d like to create an account, you can sign up here.