Hey mate 👋. Have you seen a yarn.lock file? It can be package-lock.json, pnpm.lock or npm-shrinkwrap, it does not matter that much. What matters - this file have a purpose.

package-lock.json is automatically generated for any operations where npm modifies either the node_modules tree, or package.json. It describes the exact tree that was generated, such that subsequent installs are able to generate identical trees, regardless of intermediate dependency updates.
– A manifestation of the manifest

In short - lock files represents and store information about dependencies other than your direct ones, and particular version of such indirect dependency to use.

Devon Govett

@devongovett

Lock files solve one set of problems by trading them for another set of problems. Time for a new approach.

19:05 PM - 24 Sep 2022

There is no time to explain everything in the details, let me show you the problem.

Chapter 1 - understanding the problem

Lets create an issue

Many of us used lock files for years and used them without any troubles. Everything seems good, but there is one nuance we need to discover. A little feature we are going to reveal. Let's run an experiment.
I will use yarn to demonstration purposes.

The experiment itself is for demonstration purposes only. Just blindly follow the steps or magic might not happen

• Step 1. Create empty repository.
  • yarn init and "install" it - yarn
  • you should get an empty yarn.lock file and empty node_modules folder
• Step 2. Let's add first dependency. Let it be tslib@2.0.3
  • yarn add tslib@2.0.3
  • that would create a record in yarn.lock like

tslib@2.0.3:
  version "2.0.3"
  resolved "https://something-something/tslib-2.0.3.tgz"
  integrity sha512-long-and-ugly==

• Step 3. Let's massage our environment and manually amend package.json and yarn.lock to use tslib:^2.0.3
  • in package.json: "tslib": "^2.0.3"
  • in yarn.lock: tslib@^2.0.3
  • then run yarn
  • ⚠️ it is important to have the following record in yarn.lock, which means "tslib@2.0.3 used for tslib:^2.0.3"
    • 🤔 the same will happen if 2.0.3 is the latest version available. Well, it's not the latest now, but was in the past
    • 💡 please remember this moment as a direct edit of the lock file

tslib@^2.0.3:
  version "2.0.3"

• Step 4. Add another dependency depending on tslib
  • yarn add focus-lock (v0.11.2). This package depends tslib:^2.0.3😉. Perfect match.

let's verify - our yarn.lock should contain only two dependencies - focus-lock and tslib

So that's it. We have created a problem 😎. You don't see it? Yep, the real issue is that nobody sees this problem. But it's already here.

The problem is - we have two dependencies "perfectly" matching each other.

Is it a problem?

😉 Let's perform one more experiment.

• Step 1: remove tslib
  • yarn remove tslib
• Step 2: check your yarn.lock, and it will be unchanged, as tslib is still used by focus-lock
• Step 3: shrug 🤷‍♂️
• Step 4: remove yarn.lock
  • run rm ./yarn.lock
• Step 5: reinstall packages
  • run yarn

Now check your lock file 😉. It will contain the same tslib:^2.0.3, but this time it will be resolved to 2.4.0 (the last version available by the time of writing this article)

tslib@^2.0.3:
  version "2.4.0"

You might not get the joke, let me rephrase it

😳 Existing dependencies and the order of their addition affects the shape of the end artefact 😬

• In the first experiment focus-lock reused preexisting tslib
• In the second one there will no such constrain and it used the "freshest" one.

Read it another way - any new dependency you will try reuse what you already have. Any dependency you update will try to reuse what you already have. Your's legacy, your's past, your's yesterday affects your tomorrow.

As an example - nextjs installed today and the same version of nextjs installed tomorrow can have at least different version of caniuse, as it released on a daily schedule.

But that is just a half of the problem. The real one - you can still have unexpectedly obsolete packages and hear me out - focus-lock by itself is using and is being tested with tslib 2.3.1, declaring a version below as a dependency because "it does not need any newer". Or needs, but have no idea about it, because cannot test itself agains other versions of own dependencies.

Yep, this is exactly how I broke someone one's project - import error: '__spreadArray' is not exported from 'tslib'

Another example maybe?

The example above was a little synthetic. Ok, VERY synthetic. Frankly speaking it took a while for me to find a way of reproducing such behaviour in a short and sound way, as usually everything was working well. Without direct edit hack, the one I've asked you to remember, issue will be not created.
But what would be different?

Let's redo everything from scratch, but skip the hack. Depending on particular actions and events it can result in

tslib@^2.0.3, tslib@^2.4.0:
  version "2.4.0"

or TWO versions of tslib, which can be collapsed into one via deduplication and there is another story about it.

Our problem is a part of duplication one. About the same version fragmentation and bloating node_modules with, but more often without a reason.

You might have already A LOT of duplications in your lock file - try to dedupe them if you never tried and tell me the results. At least check how many tslibs you've got.

PS: deduplication is available for any package manager, as a built-in or as a separate package. Just google.

Devon Govett

@devongovett

The way package managers like npm and yarn work seems totally backward for frontend development. They will happily install 10 different versions of a package to satisfy semver ranges, bloating bundles.

There should be a mode to fail on install for conflicting versions instead.

19:34 PM - 12 Sep 2022

A real life example?

🤔 probably the last example was also now quite sound. Let's create something better. Something that has bitten me personally multiple times.

Let's talk about a real stuff, about consuming intertwined packages.

1️⃣ cool-widget uses cool-form uses cool-button (and other packages)
2️⃣ cool-widget uses cool-button as well 😎

Again - this case is not applicable to simple projects. This case is not very applicable to monorepos.
This case requires a presence of some flexibility and a non-zero distance between package. It needs a lag, it needs a gap, it needs a latency.

This case needs one repo consuming another to have a gap in the npm logistic layer in between

Let's create a supply chain disruption ⬇️

• 1️⃣ Imagine you have a Button. Just a button exported as a npm package - cool-button. Every UIKit has such primitive.
• 2️⃣ and then you use that cool-button in many other packages. For example in cool-form and cool-widget
• 3️⃣ and cool-widget uses cool-form
  • It is a big organism consuming other molecules(form) and atoms (button).
• ...
• ➡️ And then you introduce a new major version of your very cool-button 🙀.

The tricky part here is how "major version update" propagates through packages:

• cool-button should have get a major version bump due to public interface change
• cool-form might got a patch update, as nothing changes for it's public interface. Button is implementation details and is not affecting look-n-feel of a form
• cool-widget actually does not have to do anything.... or haven't updated all required dependencies yet.

Our cool-widget might be patch-updated, as it does not change public interface, just internals, and

• it may use a new version of a cool-button, because some renovate-bot (or a button's developer) updated it in all consumers.
• depend on a cool-form, which also was just updated to use a new button, but... 😉 there is no reason for cool-widget to update dependency on cool-form as nothing actually was changed. It's the same cool-form.

In other words - dependency `cool-form` will be kept as is

What 😕? How so? Why not updated?

Let's stop here and check if cool-widget have to react and incorporate every single move of every single piece. Any reason?

🤷‍♂️ Why someone should be bothered to bump every dependency in package.json every time every dependency, a little spare part of a bigger whole, updates? It could be ten times a day for ten packages, and it sounds like a lot of useless work! Especially some work is required to perform update, run test, verify the result, deploy the result. Oh

Ok, ok. Let's imagine you are fine with it, You want it. So a button update will cause:

• 1️⃣ a cascade update of ALL packages depending on it,
• 2️⃣ and then of all packages depending on just changed
• 3️⃣ and then of all packages depending on just changed
• 4️⃣ and then of all packages depending on just changed
• 5️⃣ should I stop here?

It actually can be a loop - sometimes packages might depends on previous versions of themselves. Like babel uses babel to compile self.
So 😕 I really hope you don't want that. You probably do update in batches, or not really doing it at all, and that's ok - if stuff does work, just don't touch it.

Long story short - cool-widget have got a new cool-button@v2 directly updated, but still have the old cool-button@v1 from the kept-as-is cool-form

So now we have two buttons, with different major versions, and this cannot be deduplicated.

😠: "HEY! Wait a minute. I've got you and I am promising to be a good lad and keep my deep dependencies up to date!"
😿: "Unfortunately this is not about you, this is __about packages created by someone else, about the packages you don't control"
😕: "I dont control?"
😾: "You don't"

You have got a problem with stale intermediate dependency. A dependency of your dependency which you don't directly control. And a dependency [of your dependency of your dependency]. And a dependency [of your dependency of your dependency of your dependency]. And ....

And that is the problem we were chasing all this time.

First time this issue was encountered back in 2017 – Yark: How to upgrade indirect dependencies?

Chapter 2 - Solving the problem

It's an easy job to point a finger on a problem and tell everyone - "People! Here is the problem".
It is completely another job to propose a solution.

Going back and upgrading

Lets make another step back and look on what you do control - the direct dependencies:

• you can declare dependencies you need and their versions
  • as their versions might represent a wide range - the particular versions to use will be stored in your lock file
  • if by any reason you get a newer version of already existing dep - it will be automatically raised to the highest one. If not automatically, then after dedupe.
  • 😅 so by installing a new dep you can change dependencies of your existing?
    • This is why changes in yarn.lock might need a review
• you can manage and especially update those deps.
  • use yarn up, or yarn upgrade-interactive
  • or use a little bit more advanced npm-check-updates
  • in both cases only package.lock gonna be changed

This is how you can control direct dependencies. The ones explicitly declared in your package lock. You have no control over indirect dependencies totally derived from requirements of the real ones.

Package managers are trying to preserve as much as possible in order to reduce the impact of the change and amount efforts users might need to perform in order to manage consequences. Known as Principle of least action.

Package manager's job is to keep things. Our goal is to _loosen some bolts...

Please welcome...

Yarn-unlock-file

yarn-unlock-file is a new library, capable of editing your lock file and addressing issues created by indirect dependencies

The simplest use case is nothing more than

npx yarn-unlock-file all
# and don't forget to
yarn

That is it - the command will:

• 🔐 keep any of dependencies declared by you
• 🔓 "unlock" all indirect ones
  • "unlock" means "delete" old records from yarn.lock letting the newer, might be more correct versions to be installed

What you are dependencies?

There is one thing we forgot – the purpose of package managers. What is yarn, what is npm and what they do...

PackageManagers are primarily managing logistics - they provide transport layer between npm repository and your project.
Then PackageManagers are in charge to recreate the same environment, the same combination of packages across different machines to make our live more predictable.

However, if you use a library, you probably rely on it to be tested and behave well, but this is possible only if the library was assembled in your projects in exactly same way it was built and tested in the place of the origin - it's own space.

So it's important to let free artificial boundaries preventing the similar environments

Everything has limits, and a perfect tool capable to capture and freeze your decisions in time might need a hand.

No tool should change your decisions and change package versions picked by you, but it might help you manage decisions you haven't made consciously - concrete versions for your indirect dependencies.

Try commands like

• npx yarn-unlock-file dev to update only devDependencies, or
• npx yarn-unlock-file matching @material-ui/* to update dependencies of material-ui but not MUI itself

Safety first

There is a reasons why we are not just deleting yarn.lock - the result can be unpredictable. Or it can be "too much" - you will get a broken build or a broken application if your tests are not great in detecting anomalies.

• One should bumping your dependencies regularly, but it's not always safe and can go unpredictable in large projects.
  • One might update versions in the same range, and "unlocking" is a good way to do that.
• One might still be cautions about updating direct dependencies, or even dependencies of dependencies. But not many layers is there?
  • npx yarn-unlock-file levels all will tell you how much.
    • I haven't seen more than 8.
    • I haven't seen less than 3.
  • 😎 updating dependencies of dependencies of dependencies should be safe
    • npx yarn-unlock-file all --min-level 3

Dry run

Well, why one should blindly trust some tool to delete only what you want. Trust, but verify. Use dry run mode.
npx yarn-unlock-file all --min-level 3 --dry-run

The picture above is a good indication of how many layers can be hidden inside a simple solution, memoize-one in this case.

One might ask the question - why level 8 dependency is-arraysh is here, and what causes it?

• lets ask: yarn why is-arrayish > nyc#test-exclude#read-pkg-up#read-pkg#load-json-file#parse-json#error-ex#is-arrayish
• let's update only this branch: npx yarn-unlock-file matching nyc --dry-run

1️⃣ look 2️⃣ pick min-level you are comfortable with 3️⃣ wipe

Note about Npm-check-Updates

ncu has a "doctor" mode to update dependency, run tests, and only then update another dependency.
While this works just great - it actually can tangle your lock file beyond imagination because packages you install today affects the shape of package you install tomorrow 😉

Having more control over this process is quite beneficial.

• For years I was giving an advice of "just find something in your lock file and delete".

• For years I was looking into yarn.lock updates to correct result of deduplication sometimes by selecting a few hundred of lines to delete and regenerate.

🤷‍♂️ enough is enough. It's time to understand that deterministic builds is one problem, but supply chain disruption is different.

Authors of packages and tools you use expect and test their creations using some know (to them) set of other packages and tools, which are in turn rely on some other set of libraries. Any dependency specified within an accepted "range", and most of them are, is a subject for version mismatch. Or an expectation that the end consumers will use a new patch version of a tiny library with something very important fixed.

There are dependencies you know, the ones you want to use.
Everything else has to be unlocked on a weekly basic.

You can use Renovate, Dependabot or npm-check-updates

Sitnik the Developer

@sitnikcode

Wrote for evilmartians.com a script to auto update caniuse-lite by CI.

Every month it will send PR with new version. No more warnings in CLI and faster JS bundle without unnecessary polyfills.

Copy to your project.

gist.github.com/ai/14b7c0990a2…

13:01 PM - 12 Sep 2022

But it's not enough
https://github.com/theKashey/yarn-unlock-file

PS: Did you know that dependabot can update dependencies in your lock.file? Like here it bumps terser changing only lock file, because it's an indirect dependency.

So what?

Just try it

npx yarn-unlock-file dev
yarn

And then we will see 😈

💡: this is a first, partially experimental solution, currently capable to handle only yarn(v1 and v3).