Simplifying my old event bus architecture with Bull

The why

So before I posted about an event bus with AWS SNS and SQS. I think it works really well and it's pretty cool. It did have some rough edges that bothered me though.

• There was a lot of manual setup in production.
  • There are scripts etc that can do this but I was having to make the queue and SNS topic and link them for every new queue I wanted to make
• Local dev was messy
  • There are fairly solid Docker images that imitate SNS and SQS but the setup in code felt very messy.
• Our task worker code, while cool, felt fragile.

So how does bull help?

Task worker management:

They have a solid interface for registering and running task workers. Making it much less work on our end to handle that part of the code.

Keeping all the code contained:

All pieces of the flow exist in the same codebase. Queue creation, event passing and task running without external setup for production.

Less moving parts:

In the previous iteration you had SNS, SQS, custom task handler code, and server code. The updated version uses just server code, Bull, and Redis. While only one less system is used in the updated version. We reap many benefits from the battle tested, and frankly much better, Bull code.

Portability:

Our task code now only relies on Redis. This makes it much easier to move to another cloud provider. No code changes required.

The How

So it's pretty simple to implement. (You can refer to the Bull documentation for most of this as well.)

Setting up queues

We make a queue for each type of work we want to do. Such as emailing or sending texts.

const emailQueue = new Queue('email', 'redis://127.0.0.1:6379');
const smsQueue = new Queue('sms', 'redis://127.0.0.1:6379');

Then we have all our queues in an array so when an event comes in we can pass that to all our queues

const queues = [emailQueue, smsQueue];

function onEvent(name, data){
    queues.forEach((queue)=>{
            queue.add({
                name,
                data
            })
    })
} 

And we can define our queue processors.

emailQueue.process(emailTaskWorker);

Pretty simple stuff.

Final thoughts.

Why pass events to all queues? It allows any any queue to decide what it wants to do.

For example. If a user commented on a post, we might want to email them, text them, or generate a push notification. If in our code we did:

server.post("/posts/:id/comment, (req) => {
    server.sendEmail(req)
    server.sendText(req)
    server.sendPush(req)
})

That would get really messy. Instead we just make a record of what event happened.

server.createEvent("user-commented-on-post", {post, user});

Now any queue that cares about that event can do whatever it wants in response. It keeps our http handler code very clean as well.