Yarn Tips, part 1

Funnily, we've had a few people opening issues on our bug tracker to report that "Yarn doesn't generate a node_modules when I run an install, something is wrong!". Then we ask them to still try to run their programs and, lo and behold, it works! This is due to Yarn 2 using a new install strategy by default, that we call Plug'n'Play.

Still, Plug'n'Play isn't for absolutely everyone. Some older projects aren't compatible with it, and some others see preexisting bugs being exposed, usually under the form of missing dependency reports.

As regular users we don't necessarily want to have to deal with all that, so Yarn still features the good old node_modules strategy. Just one setting to set, an install to run, and you'll find your node_modules back! Without the Plug'n'Play stability and performance improvements, of course, but at least you'll be in known territory and ready to pursue your work.

That being said, enabling the node_modules linker should probably be seen as a last resort option. After all, there are (many) reasons we decided to make it the exception rather than the norm. But how to proceed when working with problematic packages that don't list all their dependencies? Do we need to wait for them to fix that? Fortunately not.

The packageExtensions setting is the recommended way to fix such problems. Just declare yourself the dependencies that the relevant packages are missing, and let Yarn deal with that. I myself use it quite a lot in my projects, with for example:

packageExtensions:
  "@apollo/client@*":
    dependencies:
      "@wry/context": "*"
    peerDependenciesMeta:
      "react":
        optional: true
      react-dom:
        optional: true

And from time to time, don't forget you can send PRs upstream to solve this problem durably both for you and your community!

While a large part of Yarn 2 has been to, well, implement Yarn, we also wanted to help fix the ecosystem. After all, what good is it to build a package manager that enforces strict rules if nobody is following them? So we submitted many, many PRs, adding missing dependencies to projects that forgot them, explaining the danger, and making sure that everyone had time to address those problems before our release in January.

Still, even projects that actively took steps to fix those issues sometimes had regression. For example, large OSS projects like Next.js, because of their very high velocity, sometimes merged PRs that didn't list new dependencies. In the worst cases, we only noticed it when someone opened an issue on our bug tracker. Not great!

And so, after some time, we decided to start implementing our own set of E2E tests. Baked by a bunch of GitHub workflows, they run every four hours and let us know should anything awry happen in the ecosystem. Thanks to this, we help ensure that the JavaScript ecosystem not only trends towards sanity, but doesn't deviate from the goal even when we're not looking 😉

Back in v1 our cache could, sometimes, be corrupted. It was a fairly rare occurrence, but in theory it could happen. And, should this happen, it was impossible for Yarn to detect it, which would eventually lead to very difficult debug sessions.

In the v2, Yarn is much more careful, and runs a full validation pass before attempting to install the packages on the disk. The validation is really fast (I think it takes less than 2 seconds on our own repository), and it protects against various problems. For example, since Plug'n'Play can share the same cache between all projects on your machine, it may happen that you want to modify a dependency's sources during a debug session, and forget to revert it. With the integrity check, Yarn will now let you know the problem the next time you run yarn install!

One interesting thing that made node_modules installs very slow was way the installs and the cache interacted together. Back in Yarn 1, our cache contained the unpacked archives, and we copied each file into each project during installs. We used file cloning as well (using fs.copyFile), but because of the sheer amount of files, a lot of I/O still had to be performed.

In Yarn 2, the pipeline is quite different. Plug'n'Play allows us to keep the dependencies compressed, meaning that there isn't any difference between the cache and the install artifacts! Because of this, even in the absolute worst case, we only have to perform the clone operation exactly one time for each package - compared to the hundreds of times it previously was.