I implemented a queue system in NodeJS

It's common for workloads that can be processed asynchronously to increase in your application flow. This is the situation I found myself in building EmployRemotely.com (for context, this is a platform we created to help developers find remote jobs). Upon a user being registered or an advert being submitted I was performing various actions that didn't need to be performed immediately, such as:

• Sending a user email
• Distributing that advert to various channels (Twitter, Reddit etc)
• Sending data off internally (tracking systems, Slack channels etc)

All of this was being handled within the relevant APIs, and as this list of actions grew it became more obvious this wasn't the right approach to take. It was becoming more complex, error prone and potentially providing a bad experience for the users. So I decided to look for a better alternative.

To avoid any confusion with the terminology used in this article here are the differences between the terms "adverts" and "jobs" mentioned throughout.

• Advert - A job advertisement you would typically see published on the website to attract and inform candidates to apply for a specific position at a company
• Job - A task that gets pushed into a queue system to be processed at a later stage. This can be anything.

Queues

A queue system is a way of storing enough information about a particular job for it to be carried out at a later stage. Usually the main app will store this information, and push it into a queue of jobs to be processed in the future.

Some of the benefits of a queue system include:

• Distribute the workload over time
• Decoupling work
• Retry logic
• Consistency between external system integration

In my case, if I wanted to distribute an advert to Twitter and/or Reddit I would add these to a queue, which would allow me to process them independently, in isolation which is decoupled from the original application flow.

Bull

After some research I decided to reach for Bull for my queuing system. Bull is "the fastest, most reliable, Redis-based queue for Node".

Bull simply ticked some important boxes for me. It was feature rich, had a very simple interface and looked easy enough to get up and running with. Because EmployRemotely.com is not full-time for me, time was definitely an important factor.

Implementation

1. Installation

a. Ensure you have Redis installed on your local machine.
b. Install Bull into your project npm install bull --save

2. Structure.

It always helps me to understand how things are tied together when I can see a directory structure. So, I created two new directories to separate queues and jobs.

3. Create queues and processes.

Witin the /queues/distributeAdvert.js file create my queue and processing function for each job to be processed by. The Queue constructor creates a new queue that is persisted in Redis. Every time the same queue is instantiated. The first parameter of the queue is the queue name.

// src/queues/distributeAdvert.js

const Queue = require('bull');
const sendTweet = require('../jobs/twitter');
const sendReddit = require('../jobs/reddit');

const distributeAdvert = new Queue('distributeAdvert', process.env.REDIS_URL);

distributeAdvert.process(async job => {
    const { slug, service } = job.data;

    try {
        switch (service) {
            case 'twitter': {
                const response = await sendTweet(job);
                return Promise.resolve({ sent: true, slug });
            }
            case 'reddit': {
                const response = await sendReddit(job);
                return Promise.resolve({ sent: true, slug });
            }
            default: {
                return Promise.resolve({ sent: true, slug });
            }
        }
    } catch (err) {
        return Promise.reject(err);
    }
});

module.exports = distributeAdvert;

4. Adding jobs to the queue.

In my API where I would handle the advert submission and create a database entry for it. Its here I was also previously sending this off to Twitter and Reddit to be published also.

Now I can remove these requests to Twitter and Reddit and simply replace it with the queue system we've created by adding the necessary job information to the queue to be processed.

Here the job is added to the distributeAdvert queue. The job is nothing but an Object that contains the required data to process it.

const express = require('express');
const { distributeAdvert } = require('../../queues/');

const router = express.Router();

router.post('/create', checkUser, async (req, res, next) => {
    const {
        ...
        slug,
    } = req.body;

    // ...code to insert advert into database

    try {
        distributeAdvert.add({ service: 'reddit', slug });
        distributeAdvert.add({ service: 'twitter', slug });

        return res.status(200).json({
            message: 'Your advert has been submitted successfully, good luck in your candidate search!',
        });
    } catch (err) {
        return res.status(422).json({
            message: 'There was an unexpected error submitting your advert.',
        });
    }
});

module.exports = router;

And that's all that is needed.

• We've created our directory structure within the project
• We've created our distributeAdvert queue
• We've replaced requests to third parties (Twitter, Reddit etc) with code to add these jobs into our queue to be processed.

Summary

So in summary, by implementing queues, I have now:

• Smoothed out my process
• Decoupled unnecessary tasks from important APIs
• Have a less complex and more readable process in place
• Have an approach that will scale better
• Made distributing an advert to third parties more consistent

Thanks for reading.

If you're interested in following our progress on EmployRemotely.com, including what works and what doesn't, head over to my Twitter @codebytom

Sign up to our newsletter to get relevant job opportunities emailed to you weekly