Yarn and the dark future of third party NPM clients

Yarn doesn't handle the underlying NPM infrastructure with elegance — and it might never do so.

I’ve spent the last few days wrangling with Yarn errors. Our builds we’re failing in some weird and random ways — and all signs pointed at Yarn. I can give you the TL;DR; of the investigation, and it’s this: Yarn doesn’t handle upstream NPM infrastructure errors in ideal ways.

But the problem is not (only) that Yarn code is buggy — the problem is more the disconnect which exists between Yarn (the client) and NPM’s infrastructure. The errors caused are significant enough to start conversations for moving to the NPM client. But moving back to NPM raises a bigger question about the viability of third-party package managers that rely on NPM infrastructure.

The Problem With Yarn

The problems I’ve had to debug recently relate to the fact that Yarn wraps the NPM infrastructure. Yarn doesn’t host any of it’s own packages, and therefore does not have a lot of say in how these packages are served, what errors they throw, etc.

Each of the NPM CDN failure scenarios are written into the Yarn client, but if the CDN fails in unpredictable ways (such as failed integrity checks, installation of private packages or even too many published versions) the Yarn client is not appropriately handling these cases. In the best case, things like install steps fail — in the worst Yarn exits cleanly as if nothing ever happened.

So what were the issues we were facing?

False Positives On Install

Firstly, sometimes Yarn appears to hang mid way through an install. And sometimes it is (actually) hanging. But, worryingly, sometimes Yarn will exit cleanly mid install step. And in some scenarios not running a full install will work, and other times it might not — giving you false positives on builds.

These false positives have been happening for a while throughout Yarn’s history. A quick Google shows these types of issues being raised right back to right back to 2016. But have been dismissed by Yarn maintainers as trivial “internet issues” for instance. And will presumably come and go based on NPM’s availability. But interestingly enough, the NPM status page reports do not correlate with issues seen in Yarn.

Half Downloaded Files

Secondly, whilst sometimes the errors cause the Yarn client to exit as above sometimes the NPM infra fails in different ways, such as closing connections early. Which leads to the following type of error which points to an “unexpected end of file”.

Annoyingly, both the errors do not direct your attention to the NPM CDN, but instead send you on a rabbit hole thinking the problem is on your end.

The plight of Yarn

Maybe right now you could be thinking: “Okay so Yarn has some kinks, but so does all open source — why not make a contribution and be done with it?”

But the problem goes deeper. And my concern extends more to the inherent relationship that exists between Yarn and NPM. Let me explain…

Yarn Dances to NPM’s Tune

We need to remind ourselves that Yarn is only a client wrapped around the NPM infrastructure. Because NPM holds all of the package infrastructure it makes Yarn susceptible now (and continually) to any upstream issues with NPM. Which means that Yarn will always lag behind NPM on adopting any necessary client changes that are based on CDN changes.

Yarn Is Being Ignored

To add to these difficulties that faces the Yarn ecosystem, it doesn’t help that important players such as Github are choosing to prioritise the NPM client instead instead of the Yarn client.

Yarn 1.0 Is Being Deprecated

And lastly on top of the CDN issues, Yarn 1.0 is being mostly left in the dark so that contributors can work on 2.0. But no amount of features in yarn 2.0 is going to fix the disconnect between NPM and the Yarn client. For instance, if you look at the contribution graph of the current Yarn project.

And compare to the Yarn 2.0 repo.

You see what I mean? The shift in attention only exacerbates the problem. Fixes aren’t made as quickly or as readily into the Yarn client. And these fixes might help to lessen the pain of the errors caused by NPM.

The Fix(es)

Whilst these issues are well out of your, or my hands there is a few things you can do to fix, or lessen the pain you might be feeling.

Fix 1: Use latest node and NPM

The first thing to check is ensure you’re running latest. Running latest ensures that you pick up any additional error handling scenarios built into Yarn.

Fix 2: Validate your installs

Since sometimes the installs fail midway through, you should manually ensure that your install has the packages you expect. Yarn has a util built in to do this which checks the current package.json against the node_modules. To run the command, run: yarn check --verify-tree

Fix 3: Hard Install

Another trick is to ensure you’re doing a full install by passing the --hard flag to Yarn to force a full update.

Fix 4: Swap to NPM and NPM CI

And last but not least — should all your other efforts fail — you can swap to NPM. Swapping to NPM won’t fix any CDN flakiness, but it will likely lead to better error handling for edge case scenarios.

Working Around Yarn Limitations

And that’s it. I wanted to share with you some of the difficulties that we’ve been having with Yarn, the reasoning and the potential fixes. Sadly though it raises interesting questions about the future of third-party clients that work with NPM infrastructure.

It seems without some changes to the way the ecosystem works that third-party clients are doomed to have a second-rate experience. Maybe they can fight back with better features? We can’t predict the future, but hopefully now you can at least fix your build system for now!

Lou is the editor of The Cloud Native Software Engineering Newsletter a Newsletter dedicated to making Cloud Software Engineering more accessible and easy to understand, every 2 weeks you’ll get a digest of the best content for Cloud Native Software Engineers right in your inbox.