It works on my machine

When Webpack is broken, but only in the packaged build — or how I created my first Webpack plugin

First, this story requires some background and a basic understanding of the architecture of the main project I’m working on, Foreman. Foreman is a Ruby on Rails application, used for infrastructure management. It has many plugins that add various functionality, which are written as Rails Engines. Since this application is used to manage critical infrastructure, it is installed on-premise and is shipped as packages — both .deb and .rpm. Many of the various plugins are also packaged and shipped as .rpm and .deb files.

About 2 years ago we started modernizing our application’s front-end. To achieve this, it meant integrating Webpack and npm into our assets build process. This allowed us to use cool new technologies such as React and ES6 in our client side code, as well as start to solve long standing issues we had with our legacy JS code — such as lack of linting and unit-tests. The initial step of creating a Webpack build process took a while. One of the reasons is that RPM package builders must run in a disconnected environment, so we had to figure a way of providing all the node modules to the builder as RPMs at build time, but that is a story for an entire post on its own. Once that was solved, we started gradually migrating or replacing the legacy code-base with new, Webpack-managed code.

A major challenge we had recently was providing plugins with a way to leverage the existing npm and Webpack stack in Foreman core for writing their own client-side code. We didn’t want to force every plugin to create its own build pipeline, and we didn’t want to have multiple copies of every library included in each plugin’s JS bundle. This was solved with some clever engineering work by some of my colleagues. The solution involved compiling the Webpack assets for both plugins and the core application from the main application’s code base, while providing plugins with a vendor.js file that contains multiple shared libraries — such as React, Redux, Lodash and more. This worked perfectly fine during development, and even when compiling assets for production locally.

However, when we tried installing the nightly packages with plugins, something strange started happening. The core pages which included Webpack-managed assets worked fine, but the plugin pages requiring use of the plugin’s Webpack assets didn’t load. Looking in the console, it didn’t look promising:

TypeError: e[t] is undefined

Now, part of the Webpack build process includes minifying the js code, making it smaller to download but much more difficult to debug. Luckily the browser developer tools knows how to prettify the code, so it is at least semi-readable. This pointed to the error being in the following function, on line 9:

Well, that’s not much better… but at least .exports hints at what this does. Comparing to the unminified and looking at the code around this function led me to understand that this is part of the Webpack code that handles module exports and makes them available to other modules, such as the plugin. But what is e[t]? And why is it failing here?

Since I know the failure occurs when e[t]===undefined, I set a conditional breakpoint on this line in my browser’s developer tools and reloaded the page. Now I could finally see what is happening here. e seems to be an object mapping hashes to functions:

» e
{0: ƒ, 00bb67fca9d5ae35c4aa: ƒ, 0139edb80f5e8e5773b7: ƒ, 01637e90596e3c444fa8: ƒ, 01d23de99989e6fe314f: ƒ, 01f59de879b1bcfbb8e7: ƒ, …}

And t is a hash — 2278734bfcaa57409dda in this case. But for some reason, this specific hash isn’t included in the e object. Digging inside the plugin’s minified code, I searched for the hash to try and figure out what it meant. This allowed me to make a guess at which module this hash should map to, but surprisingly, it was a module that should have been included inside vendor.js! So what happened here? Where did this hash come from? why was this working fine in the development environment? Looking in our Webpack config file, I found the following:

Git history indicated that HashedModuleIdsPlugin was introduced recently, in an attempt to allow plugins to use modules by creating a stable identifier for each module. Apparently, the default Webpack configuration generates a sequential ID for each module, making it difficult to use from inside plugins as the module IDs would change every time a module was added or removed. This explained why the hashes were only present in production, but not why this worked when compiling the Webpack assets for production locally and failed when using the bundles produced by our RPM builders. I needed to understand how this hash was generated.

Reading the Webpack documentation for this plugin, I found out that:

This plugin will cause hashes to be based on the relative path of the module , generating a four character string as the module id.

This led me to suspect that for some reason or other, when generating the Webpack bundles on the builders, the relative paths to the node modules were different for the plugins to those that were used when building Foreman core. However, from hashes this theory was very difficult to confirm. I needed some way to find the relative path that Webpack was using to generate the hash.

After some research, I found there is a plugin that does just that — NamedModulesPlugin. For some reason, the documentation (that has been deleted recently) claimed it should be used only in development and that it will show the relative path to the modules only when Hot Module Replacement is active. Looking at the code and trying it out, though, seemed to indicate that this is exactly what I needed to verify my hypothesis. When I replaced HashedModuleIdsPlugin with NamedModulesPlugin, the hashes were replaced with relative paths, both in the object mapping module identifiers to functions and in the code calling them. Locally, this continued working just as before with plugins, but the real test was when running with plugins built as RPMs.

Once the change to NamedModulesPlugin was merged into the project, it was time to rebuild the packages with the change and see if and what failed now. It took a couple of days to get the packages built again, since in the meantime some unrelated changes were merged that led to broken builds. But once they finally were, the moment of truth arrived — was my hypothesis correct?

Unsurprisingly, the same e[t] is undefined error showed up again. Only difference is that this time t should point to the actual module that was failing to load, so we can finally figure out why this was failing. Returning the conditional breakpoint to the vendor.js code allowed me to find what it was:

» t
"../../../../../../../usr/lib/node_modules/react/index.js"

React? That’s odd! We use React in multiple pages in the core application as well, and they work just fine, so it must be available in vendor.js! Quick CTRL+f for react in vendor.js led to identifying the culprit. This is the identifier that was present in vendor.js for the React module:

"../../../../usr/lib/node_modules/react/index.js"

Bingo! Notice the different number of leading ../’s compared to the identifier the plugin was looking for. This indicated that indeed, during the build process, Webpack was being run from different paths when building the plugin, compared to when building Foreman core.

Now that we finally got to the root of the issue, it was time to fix it. There were two different ways we could go about this:

1. Make sure that the build always occurs from the same root directory, leading to consistent module IDs.
2. Create a unique identifier for the modules that doesn’t care about where the build was started from, only about the module itself.

Option 1 could possibly work, but it required some deeper digging in our fairly complex build pipeline, followed by changes that may lead to other unexpected issues further down the process. It would also tie the whole build process to the Webpack configuration and reduce its flexibility in the future.

Option 2 required finding a way to make sure that module IDs are consistent across builds, regardless of what folder the build process was started from. One idea that was brought up was to use a hash of the module’s entire source code, but that would mean that every time any module is changed, all plugins would have to be rebuilt — since the hash would change as well. While for some changes it makes sense to also have plugins updated, usually module APIs don’t change much between versions, so that a minor change shouldn’t matter to the consuming plugins.

The solution that was finally decided on was to strip everything up to node_modules from the path for the module — so that React, for example, would get an ID of node_modules/react/index.js instead of something like ../../../../usr/lib/node_modules/react/index.js. We don’t really care where the node_modules folder is, and it shouldn’t matter to plugins either. This ID will remain consistent across builds, and for as long as the module doesn’t change its internal structure. But how can we do this?

To figure this out, I looked at the source code of the NamedModulesPlugin that is shipped with Webpack 3 (the version we are currently using). The code was surprisingly short — a loop over all modules, setting the ID using libIdent(). Some searching inside webpack code allowed me to understand the this function call will return the path to the module, without trying to dig to much into its internals.

Since the plugin doesn’t provide any means of modifying the ID, what was needed now was to make a new Webpack plugin that strips everything up to node_modules:

This code is almost the same as the original NamedModulesPlugin. The only change I added are lines 14–16, which strip out the extra part of the path from the module ID where needed.

As I assume others may run into this or similar issues in the future, I have extracted this plugin from our project to its own repository, and released it as a node module.

While this story was mostly written in the first person, as it depicts my perspective of the events, it is important to note that this solution, like everything I work on, is the result of a team effort.

Special thanks go to all those who helped in the process of fixing this issue, in no particular order: Daniel Lobato Garcia, Ewoud Kohl van Wijngaarden, Eric Helms, Avi Sharvit, Ohad Levy, Walden Raines, and anyone else who lent a hand and I forgot to mention.

This solution would also not have been possible if Webpack hadn’t been an open-source project — as I would not have been able to dig into its internals, extract and modify code from it otherwise.