The other day I caused a build failure that prevented my teammates from merging in their respective PRs. It didn't surface until I merged my PR.
Or so I thought.
There are two reasons I want to share this experience:
1) Share my thought process to help you solve a similar problem
2) Destigmatize breaking things at work because it happens
And I suppose also because our memories are fickle.
Debugging Adventure Begins
It started with this build failure in our pipeline Slack channel:
// truncated
yarn workspace v1.xx.x
yarn run v1.xx.x
Error: ENOENT: no such file or directory, scandir
'/app/your-package/dist'
The first thing I had to go off of was the thing that is supposedly missing, which is the dist folder. The dist folder contains all code meant for production and it is generated by running the related build script.
To run the build for production in this package, we could
1) yarn run build:production (or whatever is in your package.json)
2) yarn workspace your-package run build:production (in Dockerfile)
Now that I had a better sense of what the thing was that was missing, I moved onto reproducing the issue and there were two ways I could do that.
I tried running the production build locally first.
It succeeded. Which lined up with the fact that I didn't see any issues when I was the testing feature changes for the application locally.
Next, I tried building it again in my QA box.
The error showed up!
I tried 3 different things based on the fact that this was my first time introducing a new shared package in a monorepo. I figured I probably missed a couple steps. So I tried to copy pasta exactly from a similar PR. But nope.
Still failed so I dropped those commits.
It was at this point that I decided to search through our company's Slack for my error message to see if someone else had run into a similar issue.
Turns out, someone did. 1 year ago.
Apparently, when you suddenly decide that a package in your mono-repo needs to import another package in your mono-repo, you need to make sure you explicitly list the import package as a dependency in the package.ison
I felt kind of silly reading that. It felt so obvious.
But the underlying reason why is really the kicker. And it really took me back to my time on the devops side of our company a couple years back.
When you run your package locally, assuming everything is up-to-date, all the dependencies you need should be available. So when I ran my feature branch locally, I didn't see an issue for that reason. I had everything I needed.
However, a best practice for Docker builds is to not install unnecessary dependencies in order to save time and build size. So unless you explicitly list a package as a dependency, it will not be installed when Docker runs 🐳
Since my QA box runs a docker build each time we build the box, it should have shown me the error while I was testing. Turns out the reason I didn't see it is because I happened to gloss over the error line (it visually wasn't obvious there was an error aka no glaring red colour and no alarms) and I had coincidentally observed that my feature was working as expected on the box.
After testing my fix, I updated my teammates and all was well.
Lessons Learned
1) I'm curious if other developers would have searched in Slack sooner than I had. I should get into the habit. I tend to Google first. Unless it feels like a problem specific to my company i.e. cannot run package locally.
2) If you have AFK time that day, let your team know so people can tag in.
3) What's still unclear is whether the error message I encountered was just a standard cryptic one or if our error handling could be better somehow...
Not a lesson. But remember to celebrate your growth. I think earlier in my career, I probably would have had a tough time solving this without pairing. And while there's nothing wrong with pairing, I do use my ability to solve more complex problems independently as a measure of growth.
So here's to growing and sharing. Learning in public 🥂
Keep it candid,
Thuy 🙋🏻♀️
Top comments (0)