Run a worker alongside Next.js server using a single command

#nextjs #javascript #api #worker

By default Next.js only has one entry point: the web server, sourcing /pages. But if you're building a real API, you may need other entry points to run scripts and/or run a worker to process background jobs.

You could just add a worker.js file and execute it with node /path/to/worker.js but you'll lose ES6 imports and therefore compatibility with your helpers. No point in duplicating the Next.js build stack, let's see how we can reuse it.

Next.js allows us to extend its Webpack config in next.config.js, we only need to specify our new entry points there. As stated in my previous article Build a full API with Next.js:

const path = require('path');

module.exports = {
  webpack: (config, { isServer }) => {
    if (isServer) {
      return {
        ...config,
        entry() {
          return config.entry().then((entry) => ({
            ...entry,
            // adding custom entry points
            worker: path.resolve(process.cwd(), 'src/worker.js'),
            run: path.resolve(process.cwd(), 'src/run.js'),
          }));
        }
      };
    }
  },
};

Pretty basic. But how do we run them? node ./src/worker.js won't work because it needs go to through Webpack. So we have to wait for the file to have compiled with next dev or next start commands. Once your app is built, the compiled file will be available at .next/server/worker.js so we can basically just run node .next/server/worker.js and now it'll work!

But that's a poor developer experience, as we have to wait for first compilation before we run our worker process in a second terminal. To run the worker alongside the server with a single command, I rely on:

npm-run-all to execute multiple commands in parallel,
wait-on to wait for the file to exist before running the worker,
nodemon to reload the worker on file change.

Here's how my package.json looks like:

{
  //...
  "scripts": {
    "dev:app": "next dev"
    "dev:worker": "wait-on .next/server/worker.js && dotenv -c -- nodemon .next/server/worker.js -w src/server -w src/shared"
    "dev": "npm-run-all -p dev:worker dev:app",
    "worker": "dotenv -c -- node .next/server/worker.js",
    "script": "dotenv -c -- node .next/server/run.js script",
    "job": "dotenv -c -- node .next/server/run.js job",
    //...
  }
}

A few notes here:

I'm only watching back-end utilities with nodemon (src/server and src/shared) so front-end changes don't unnecessarily reload the worker.
I use dotenv-cli to source .env files because Next.js won't inject them in custom entry points.
Running a script or a job is managed here by a single entry point run.js but you could have 2 separate files to handle this. As it's an on-off process, I don't feel the need to use wait-on nor nodemon.

Hope this helps!

Top comments (2)

Max • Dec 22 '22 • Edited

great article, thanks! Although in next 13 I had to use a little bit different config:

if (isServer && config.name === 'server') {
      const oldEntry = config.entry

      return {
        ...config,
        async entry(...args) {
          const entries = await oldEntry(...args)
          return {
            ...entries,
            'worker': path.resolve(process.cwd(), 'worker/index.ts')
          }
        }
      }
    };

That's because after the server entry point there's also an edge-server entry point, and if we overwrite the entry() function it breaks all the entry point that come after the server one.