delay.run

Posted on Nov 14 Edited on Nov 14

Case study (hypothetical): the customer who paid, but the app still said 'Free plan'

#webhooks #backend #api #architecture

Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run

This is a hypothetical case study. The product, the numbers and the timeline are invented; the bug is one of the most common ones in subscription billing.

The setup

A small SaaS app sells a Pro plan through a payment provider's hosted checkout. The integration was built the way many tutorials show:

The user clicks "Upgrade", and the backend creates a checkout session.
The user pays on the provider's page and is sent back to the app.
The provider sends webhooks. The app's handler listens for subscription events and sets the account's plan from the subscription status in the payload.

The handler was short:

app.post("/webhooks/billing", verifySignature, async (req, res) => {
  const event = req.body;
  if (event.type.startsWith("customer.subscription.")) {
    const sub = event.data.object;
    const plan = sub.status === "active" || sub.status === "trialing" ? "pro" : "free";
    await db.accounts.update({ billingCustomerId: sub.customer }, { plan });
  }
  res.sendStatus(200);
});

It returned 200 quickly and processed events in a background queue with several workers. Everything about it looked reasonable.

What customers saw

Roughly 1 in 150 upgrades, a customer paid, got a receipt email from the provider, came back to the app, and saw "Free plan". Refreshing didn't help. Waiting didn't help. They contacted support, who would click "Sync from billing" in the admin panel, and the account would flip to Pro.

Support assumed it was a delay. It wasn't.

What actually happened

A new subscription produces several events in quick succession. For a card that needs confirmation, the sequence is typically:

customer.subscription.created with status: incomplete (payment not yet confirmed)
invoice.paid
customer.subscription.updated with status: active

The provider sends these within a second or two of each other, and it says clearly in its documentation that it does not guarantee delivery order. Deliveries can also be retried, so an older event can arrive after a newer one.

On top of that, the app's queue had several workers. Even when events arrived in order, two workers could pick up created and updated at the same time, and whichever finished last won.

So occasionally the app processed updated (active) first and set the plan to Pro, then processed created (incomplete) a few hundred milliseconds later and set it back to Free. The handler had no idea that the second event was older. It applied whatever status the payload contained.

The admin "Sync from billing" button worked because it ignored webhooks completely: it fetched the subscription from the provider's API and used the current status.

Need webhook delivery you can actually audit? delay.run retries failed HTTP callbacks and keeps full run logs, so a missed payment event doesn't silently leave a customer on the free plan.

Why the obvious fixes weren't enough

"Process events in order." A single worker with a strict FIFO queue would remove the worker race, but not the delivery-order problem, and it makes one slow event block every other customer's billing.

"Compare event timestamps." Better: ignore an event if its created timestamp is older than the last one applied for that subscription. But event timestamps have one-second resolution, and the created and updated events here were often in the same second. Ties still lost.

"Only listen for updated." Ignoring created fixes this particular sequence and breaks others. There are many paths through a subscription's lifecycle (trials, failed first payments, payment method updates, plan changes, cancellations at period end) and each emits a different mix of events.

The fix: treat webhooks as a doorbell

The team changed the handler to treat every subscription-related webhook as a signal that something changed, not as the description of what changed.

async function handleBillingEvent(event) {
  const customerId = event.data.object.customer;
  // Always read the current state; never trust the payload's snapshot.
  const subs = await billing.subscriptions.list({ customer: customerId, status: "all", limit: 10 });
  const active = subs.data.find(s => ["active", "trialing", "past_due"].includes(s.status));
  await db.accounts.update(
    { billingCustomerId: customerId },
    { plan: active ? "pro" : "free", billingSyncedAt: new Date() }
  );
}

Whichever event arrives, and in whatever order, the handler fetches the current subscription state from the provider and writes it. Processing the old created event last no longer matters, because it reads the same current state as the newer one.

Three more changes went with it:

Serialise per customer, not globally. Jobs are keyed by billing customer ID, so two events for the same customer never run concurrently, while different customers still process in parallel. Most job queues support this as a "group", "concurrency key" or "partition key".
Sync on return from checkout. When the browser lands on the success URL, the backend runs the same sync function immediately. The user sees Pro straight away, without waiting for any webhook.
Reconcile nightly. A scheduled job compares every account's plan with the provider's subscription status and fixes and logs any mismatches. In the first week it found 31 accounts, some from this bug and some from older ones.

Results

Stuck upgrades went to zero. The "Sync from billing" button was used twice in the following month, both times for unrelated reasons. The cost was one extra API call per webhook, well within the provider's rate limits for an app of this size.

What I'd take from it

Webhook order is not guaranteed. Not by the sender, and not by your own queue. Design as if every event can arrive late, early or twice.
A payload is a snapshot from the past. For state that matters, fetch the current state.
Serialise per entity. Concurrency is fine across customers and dangerous within one.
Give users a sync on return. The redirect back from checkout is the moment they care most.

Short follow-up jobs like "sync this customer in 30 seconds, and again in 5 minutes if the subscription still looks incomplete" are exactly the kind of delayed work delay.run is built to schedule, but any durable queue with per-key concurrency will do.

DEV Community