Gulp is killed because of a long-running process

I have been working on a Web application codebase that has multiple themes for different websites. When building the code during a deployment, the themes have multiple SCSS/CSS assets and scripts that go through Gulp pipes for cleaning, minifying and concatenating the code to a single build file.

Unfortunately, the build process has not been updated in years, and the theme folder structure is messed up (legacy code). So the more we added themes to our application, the longer the Gulp build process on deploy would take.

An awful 20–30 minutes on our different environments.

Yeah, production included.

Being an unacceptable situation, I analyzed how we could improve the Gulp process. Having 42 themes with multiple CSS assets and a ton of JavaScript files going through Gulp pipes, the total amount of parallel tasks where approximately 217. This for sure is not the perfect architecture for compiling assets. But in our world, nothing is perfect, so instead of doing a complete refactor of our application assets, we decided to improve the current build process.

Gulp Killed — The memory usage problem

When running long-running processes the gulp command may be killed by the server and this is annoying. Before we investigate for a solution, this is the problem that we encountered : gulp process is killed by the kernel because the memory of the server is running out.

One solution would be to add memory to the virtual machine or server. But this is not an option for us.

Let’s optimize!

Process Gulp tasks only if needed, thus, when assets have changed in our themes

He first idea that came to my mind when solving the problem was to process only the Gulp tasks that really needed to run and proceed to the next one if not. The trick would be to check if the task assets have been changed and proceed through pipes if a change was made since the last build.

Discovering existing Node solutions that did not do the trick

While searching for an appropriate solution to our problem, I discovered two Gulp plugins that seemed to be helpful. The first, gulp-changed checks against build files, for example, ES5 files generated by Babel.js from the source ES6 files. This didn’t help because we wanted the build process to be faster and not check the build files themselves.

gulp-changed-in-place was the second option, this plugin monitors source files, ES6 files for example. This allows you to do things like apply source formatting or linting only to changed files while working on them.

The second option seemed promising, although, it only sends files that were modified through Gulp pipes. When we needed to send all files if if one changed.

Complete Gulp tasks only if assets have changed with gulp-process-if-modified

With the inspiration provided by the previously mentioned Gulp plugins, I decided to code my own.

You can check it out here :

$ npm install --save gulp-process-if-modified

var gulp = require('gulp');
var processIfModified = require('gulp-process-if-modified'

Using the power of cache files and Gulp pipes, I was able to process only tasks for which assets have been modified since the last build. Every Gulp task have a cache file in which we compare the hashed content of assets and determine if a change was made.

It’s simple : if a change was made, process all assets. If not, process none.

This means that if we change a single CSS file for a theme, we won’t need to compile the 42 themes we have in our application but only the one that have modified assets (either Javascript or SCSS for example).

This decreased our build process duration from 25–20 minutes to 9–20 seconds ! 🤩

Most of the time when coding, we are motivated by existing open source code that we can adapt to our use case. In this case, I could not use the existing Gulp plugins. I did not reinvent the wheel, but did change the tire !

Let me know what you think and do not hesitate to comment on the code and provide feedback or pull requests.

Cover Image : Cris Saur (@crisaur) from Unplash