Configuring Incoming Webhook Queues in Bullet Train

Attractor is a pretty background-job intensive app. If you think about it, every commit's changes have to be analyzed, and the results stored, to keep track of code quality and emerging tech debt.

Since it's based on Bullet Train, it makes sense to handle most of these workloads in incoming webhooks. Broadly speaking, there are three groups of incoming webhooks at work:

• webhook handlers for processing code analytics data being reported back from sandbox containers,
• webhook handlers for webhooks sent from GitHub (e.g. when a new pull request is created),
• everything else, e.g. webhooks coming from Paddle, my payment provider.

Webhook Processing

Each incoming webhook in Bullet Train is processed by a controller that is created by super scaffolding.

It might look like this:

class Webhooks::Incoming::SandboxWebhooksController < Webhooks::Incoming::WebhooksController
  before_action :authenticate_token!

  def create
    Webhooks::Incoming::SandboxWebhook.create(data: JSON.parse(request.body.read))
      .process_async
    render json: {status: "OK"}, status: :created
  end

  # ...
end

The created model file contains the processing logic:

class Webhooks::Incoming::SandboxWebhook < ApplicationRecord
  include Webhooks::Incoming::Webhook

  def process
    # logic goes here
  end
end

The actual processing is done by a generic Webhooks::Incoming::WebhookProcessingJob that's part of Bullet Train and calls back to this process method. In other words, in a vanilla Bullet Train app, Sidekiq is responsible for working off your incoming webhooks.

So, let's look at the respective sidekiq.yml file:

:concurrency: 5
staging:
  :concurrency: 5
production:
  :concurrency: 5
:queues:
  - critical
  - default
  - mailers
  - low
  - action_mailbox_routing

There are 5 Sidekiq queues configured. A pretty nasty issue is hidden here in plain sight, though: All incoming webhooks are processed in the default queue.

Dynamically Configuring the Webhook Queue

Why is this a problem? Let's walk through a scenario where a new customer signs up.

1. she authenticates the Attractor GitHub app, and connects two pretty large repositories,
2. she enters her credit card information and wants to conclude the sign-up,
3. nothing happens, she is stuck.

What's happening here? It took me a while to find out.

As pointed out above, all incoming webhooks are processed in the default queue, which in this case means:

1. all analysis jobs (which can be hundreds or even thousands) are queued up there,
2. the one incoming webhook responsible for processing the payment, too - right at the bottom.

In other words, the payment would be processed only after all analysis jobs (and not only hers, maybe a couple of other customers did sign up simultaneously) have concluded. An unbearable situation, to have a customer hanging in a state of uncertainty whether her payment did successfully activate her subscription of Attractor.

To remedy this, I could have changed the sign up flow to first require potential customers to check out, then allow them to connect the GitHub app. That would have been a pretty large effort, though, so I opted to try something else.

We can reopen the Webhooks::Incoming::WebhookProcessingJob and have the queue_as method decide dynamically where to queue the webhook it's meant to process:

Webhooks::Incoming::WebhookProcessingJob.queue_as do
  webhook = arguments.first

  case webhook
  when Webhooks::Incoming::PaddleWebhook
    :critical
  when Webhooks::Incoming::SandboxWebhook
    :low
  else
    :default
  end
end

The only remaining question was where to put this small monkey patch. Placing it into an initializer didn't work, because your app's constants (such as the Webhooks::Incoming::WebhookProcessingJob class) haven't loaded at this time.

So the only option is to run it after the app has finished initializing, i.e. in an after_initialize block in your config/application.rb:

require_relative "boot"

require "rails/all"

# ...

module MyApp
  class Application < Rails::Application
    # more config

    config.after_initialize do
      Webhooks::Incoming::WebhookProcessingJob.queue_as do
        webhook = arguments.first

        case webhook
        when Webhooks::Incoming::PaddleWebhook
          :critical
        when Webhooks::Incoming::SandboxWebhook
          :low
        else
          :default
        end
      end
    end
  end
end

With this tweak, all webhooks coming in from the code analysis sandbox are placed in the low job queue, keeping others in the default one. Subsequently, when a webhook comes in from GitHub indicating a new pull request, it is processed before any potentially waiting SandboxWebhooks.

Even more importantly, incoming webhooks from Paddle are placed in the critical queue, making sure subscription changes are always processed first.

Thanks to Kasper Timm Hansen for pointing me in the right direction! 🙏