delay.run

Posted on Nov 15

Before your payment webhooks go live: a 12-point production checklist

#api #webhooks #backend #security

Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run

Webhooks from a payment provider are the moment your system finds out that money moved. In the sandbox they're easy: you send a test event, your handler logs it, done. In production they arrive twice, out of order, hours late, from a provider whose retry schedule you didn't read, to an endpoint that went down during your last deploy.

None of those are rare. Here's the checklist I'd go through before switching a payment integration to live mode. It's written with card payments and subscriptions in mind, but almost all of it applies to any provider that sends webhooks.

Receiving

1. Verify signatures on the raw body

Every serious provider signs its webhooks. Verify the signature against the raw request bytes, before any JSON parsing, using the secret for that specific endpoint. Reject anything that fails with a 400, and log the failure (without logging the secret). Test it with a deliberately tampered body, so you know the check actually rejects things.

2. Return 2xx fast, process later

The provider will time out your request after a few seconds (often 10 to 30) and retry. Do the minimum in the request: verify, store the raw event, return 200. Do the real work (updating orders, sending emails, provisioning accounts) in a background job. A handler that sends an email inline will eventually send it twice when a slow SMTP server makes the provider time out and retry.

3. Store every event before acting on it

Keep a table of received events with the provider's event ID as a unique key, the type, the raw payload, the received time and the processed time. It's your audit log, your deduplication mechanism, and the thing you'll query at 2 a.m. when someone asks "did we ever get the webhook for this payment?".

Processing

4. Make every handler idempotent

Duplicate deliveries happen: retries after timeouts, the provider's own at-least-once delivery, and manual resends from the dashboard. Deduplicate on the event ID, and also make the action itself safe to repeat ("mark paid if not already paid", not "add a credit").

5. Don't depend on delivery order

Providers generally don't guarantee order. A subscription.updated can arrive before subscription.created; a refund can arrive before you processed the charge. Either re-fetch the object's current state from the API when an event arrives, or compare versions and ignore anything older than what you have.

6. Handle "not found yet" without dropping the event

If the event refers to an order or customer your database doesn't have yet (because your own write hasn't committed), don't return 200 and forget about it. Store it and retry the processing on your own short schedule, with an alert if it's still unmatched after an hour.

7. Handle the event types you don't use

Your endpoint will receive event types you didn't subscribe to on purpose, or new ones the provider adds. Return 200 and ignore them, rather than throwing a 500 that makes the provider retry for days. Better still, subscribe the endpoint only to the event types you handle.

Checklist item: a retry policy that isn't a sleep() in the request. delay.run schedules the follow-up HTTP call with retries and a run history you can inspect.

Operating

8. Know the provider's retry schedule and disable rules

Find out how long the provider retries a failing endpoint (for some, up to three days) and what happens after that. Some providers automatically disable an endpoint after repeated failures, and send a warning email to whoever's address is on the account. Make sure that address reaches someone who'll act on it.

9. Alert on silence, not just on errors

Errors are easy to alert on. The harder failure is no webhooks at all: an endpoint disabled, a DNS change, a firewall rule, a deploy that removed the route. Track "time since last event received" per event type you rely on, and alert when it exceeds what's normal for your traffic.

10. Reconcile with the provider's API

Run a scheduled job that compares your records with the provider's: payments marked pending for more than N minutes, subscriptions whose status differs, refunds you don't know about. Webhooks are a notification mechanism; the provider's API is the source of truth. The reconciler catches everything the first nine items miss.

11. Plan secret rotation

You'll need to rotate the signing secret eventually: someone leaves, a secret leaks into a log. Check whether the provider supports two active secrets during rotation, and make your handler accept a list of secrets so rotation doesn't need a coordinated deploy at the exact right minute.

12. Test the failure paths, not just success

Before going live, deliberately:

Return a 500 and watch the provider retry, then succeed.
Send the same event twice and confirm one effect.
Send events out of order and confirm the final state is right.
Take the endpoint down for ten minutes and confirm nothing is lost when it comes back.
Rotate the signing secret in the sandbox.

Most providers' CLIs and dashboards let you resend events to make this easy.

The short version

Verify on raw bytes, store first, return fast, process idempotently in the background, ignore order, alert on silence, and reconcile on a schedule.

Several of these items (retrying "not found yet" events, debouncing re-fetches, scheduled reconciliation) are delayed or scheduled jobs. delay.run handles that kind of work, but a durable queue plus cron will do the same job.

DEV Community