Modern serverless typescript job queues in >= 2022

We’re closer to Jan 2023 than 2022. Serverless has taken over our stack, but we’re still struggling with stateful job queues for background work & scheduled jobs. We have to configure Redis, configure our queues, set up long-lived workers, and then handle observability, capacity, and failures ourself.

Why is this important? You need background jobs to have a reliable and fast system. But...

Background jobs aren’t modern, and it’s making our lives harder than it needs to be.

In this post, we’ll talk about what a modern job system looks like, how you’d use it, and then show a library that implements everything already 😎

What would a modern TypeScript based job queue look like?

• Truly serverless: we shouldn’t have to configure redis, servers, capacity, etc.
• Zero config: we don’t want to have to configure each queue before we need it.
• Fan out (event driven): you should be able to trigger > 1 job from a single call
• Fully typed: you should always know the data available to each function with full types
• Reliable: failures should be retried automatically
• Able to handle complexity: jobs are complex, and you should be able to specify a function as many individual steps for parallelism and reliability
• Kitchen sink: a good platform should also handle things like idempotency, throttling, backoffs, and scaling for you

The world has gone serverless. We shouldn’t have to worry about scaling, failures, and stateful servers. We shouldn’t also have to worry about configuring queues, or jobs failing because the data you pushed onto a queue was invalid. And we should be able to run more than one function at once!

Alongside serverless, the secret sauce is: job queues should be event-driven. Node has an event emitter built in, and here’s the basics: you write functions that listen for specific events. When you send those events, the relevant functions run. Simple! This should be the way background jobs work, too.

Designing the API

Creating serverless background jobs should be simple, with zero setup:

import { createFunction } from "inngest";

export default createFunction(
  // Function name, for observability, logs, and metrics
  "Send welcome email",
  // Event name that runs the function
  "app/user.created",
  // The function code
  ({ event }) => {
    sendEmailTo(event.data.id, "Welcome!");
  }
);

And running them should be easy:

// Send events
import { Inngest } from "inngest";
const inngest = new Inngest({ name: "My App" });

// Run the function above automatically, in the background
inngest.send("app/user.created", { data: { id: 123 } });

Events trigger any number of functions automatically, in parallel, in the background. Inngest also stores a history of all events for observability, testing, and replay.

This solves all of our problems. We can use any framework (like NextJS) to build serverless functions that deploy to Vercel, Netlify, or Cloudflare, then run reliable background functions that do anything you need:

• Sending emails
• Calling external APIs
• Run functions on a schedule to generate reports
• Anything that shouldn't happen in your core API to make your API faster and more reliable

You should never have to set up Redis, configure queues, set up SQS or SNS on AWS to get this functionality out of the box.

We've released our SDK that allows you to do this in any javascript or typescript project today, for free: https://github.com/inngest/inngest-js. You can read the docs here: https://www.inngest.com/docs

I'd love to break down how this works under the hood and this post is already quite long. Let me know your thoughts and if you want to see the architecture and system design!