Enable retries only after the webhook consumer can prove that one authenticated event produces at most one committed logistics update. Preserve evidence of the access decision. A duplicate counter cannot establish who submitted an event or whether its first delivery committed.
TL;DR: Make the consumer idempotent by event ID through a database uniqueness constraint on the authenticated source, tenant, and event ID. Commit the deduplication record, shipment update, and audit decision together; acknowledge afterward. For a Node.js backend, the decisive mechanism is the transaction boundary.
This architecture decision concerns shipment status events arriving during an outage. Access auditability is the acceptance criterion: an operator must connect an accepted delivery to an authenticated principal, its tenant scope, and the resulting state transition. All identifiers, request fields, and capacity numbers below are proposed design examples or synthetic assumptions.
What must survive a webhook delivery outage?
Three invariants determine the design. An unauthorized request cannot reserve an event ID. Two authorized deliveries cannot commit the same shipment transition twice. A successful acknowledgement means the receiver has durably accepted responsibility.
Authenticate before deduplicating.
The proposed Node.js handler authenticates the sender and resolves its permitted tenant and source before opening the business transaction. If the sender uses signatures, verify the original request bytes according to that sender's documented protocol. Bind the claimed tenant to the authenticated principal; a tenant field in the body is not sufficient authorization. Require a nonempty event ID, a recognized event type, and a shipment identifier within that tenant. Enforce request size limits. The bearer credential used later is a sandbox contract choice, not a universal webhook authentication mechanism.
Use the authenticated source, tenant, and event ID as the deduplication key. CloudEvents provides a useful envelope reference: it defines event uniqueness through source and ID. Adding a tenant boundary is an application authorization decision.
The limitation is scope. A retained event key can prevent another local commit; it cannot establish delivery order or make a remote side effect transactional.
If the database is unavailable, this synchronous receiver has not accepted responsibility and must not acknowledge success. If it commits but loses the response, redelivery must find the committed event and return the configured successful duplicate response. The distinction determines whether retries recover work or repeat it.
Decision: couple the event, transition, and audit record
Start a database transaction and attempt to insert the key under a unique constraint. Only the transaction that inserts the key proceeds with the shipment mutation. In that same transaction, record an audit decision containing the principal identifier, credential version identifier, authorized tenant, event key, outcome, and transaction correlation identifier. Commit before acknowledging. Store identifiers, never credential values.
A preliminary lookup is insufficient.
Consider two concurrent copies of the synthetic event event-demo-1042. Both handlers may read an empty lookup result before either inserts anything. A uniqueness constraint must arbitrate the subsequent insert, and the losing handler must take the duplicate branch without executing the shipment update. In the Node.js implementation, inspect the driver's conflict result explicitly: an insertion that did nothing is not permission to continue. If the chosen database aborts a transaction after a uniqueness error, roll it back before reading the committed event in a fresh transaction. If the first attempt instead rolls back, another attempt must be able to claim the key and complete the work. This is the specific trap in a lookup-then-write implementation: a reassuring log line about the lookup says nothing about which concurrent request eventually committed, and an audit record written outside the transaction can describe an update that never happened.
| Option | Failure boundary | Access evidence | Decision |
|---|---|---|---|
| Process-local event ID cache | Restart loses keys | Logs do not enforce uniqueness | Reject as the correctness boundary |
| Key committed before shipment update | Crash can strand unapplied work | Reservation does not prove completion | Reject for synchronous processing |
| Key, shipment update, and audit decision in one transaction | Requires a shared transactional store | Evidence describes the committed transition | Select for local state |
| Durable inbox and asynchronous worker | Acceptance and completion are separate | Both transitions require evidence | Select when downstream outage buffering is required |
The trade-off is coordination. The selected design is not suitable for atomically updating unrelated stores. For remote notifications, commit an outgoing work item with the shipment update and give its dispatcher a separate retry policy. Downstream idempotency is still needed where supported: losing a remote response leaves the dispatcher uncertain about whether the action happened.
A durable inbox is a valid variation when reception must continue during a shipment processor outage. Acknowledge after committing the inbox event and acceptance evidence. A worker later commits its processing status, shipment update, and completion evidence together. Receipt must never be presented as completion.
That extra lifecycle has an operational cost: someone must monitor pending work, reconcile acceptance against completion, and own replay permissions.
Exercise the critical path before enabling retries
The following curl requests exercise a proposed Node.js receiver contract. They do not implement the handler. The sandbox principal may submit events for logistics-sandbox; the receiver derives the authorized source from that principal. Both deliveries describe the same depot arrival.
Set WEBHOOK_RECEIVER_URL to your own sandbox endpoint and inject WEBHOOK_TEST_TOKEN through the normal secret mechanism. These names identify test configuration, not a platform API.
curl --fail-with-body --silent --show-error \
--request POST "${WEBHOOK_RECEIVER_URL}" \
--header "Authorization: Bearer ${WEBHOOK_TEST_TOKEN}" \
--header 'Content-Type: application/json' \
--data-binary '{"id":"event-demo-1042","tenant":"logistics-sandbox","type":"shipment.depot_arrival","shipment_id":"shipment-demo-42","depot_id":"depot-demo-7"}'
curl --fail-with-body --silent --show-error \
--request POST "${WEBHOOK_RECEIVER_URL}" \
--header "Authorization: Bearer ${WEBHOOK_TEST_TOKEN}" \
--header 'Content-Type: application/json' \
--data-binary '{"id":"event-demo-1042","tenant":"logistics-sandbox","type":"shipment.depot_arrival","shipment_id":"shipment-demo-42","depot_id":"depot-demo-7"}'
Two successful responses prove little.
Inspect committed records: require one depot-arrival transition and one committed key. Audit evidence must identify the principal authorizing that transition. Duplicate delivery attempts may have separate audit records, but must not appear as additional business transitions. These are acceptance assertions, not measured results.
Through the test harness, submit concurrent copies, terminate the handler before commit, drop its response after commit, and submit the fixture with an unauthorized principal. Require, respectively, one mutation, an attempt that remains eligible for retry, a successful duplicate acknowledgement, and rejection without reserving the key. A different event ID must remain independently processable.
Also reuse the key with changed business fields. In this proposed contract, that is an integrity conflict. Compare a stored fingerprint of normalized business fields and investigate without applying another mutation. Define normalization explicitly; transport headers and delivery-attempt timestamps must not change that fingerprint.
Finally, send shipment observations out of order. Deduplication cannot decide whether an older observation may overwrite newer state. Specify that transition rule separately.
Retain proof without retaining every byte
Access auditability requires a retention policy. OWASP's secrets-management guidance covers restricted access, auditing, and secret lifecycle management. Apply that discipline to the audit trail: retain credential identity references while excluding token values and authorization headers. Restrict access to shipment identifiers as well.
For an illustrative workload, assume 200,000 distinct events per day and 300 bytes per minimal deduplication record. At 14 days, that is 2,800,000 records and 840,000,000 bytes of record data. Indexes, row overhead, replication, backups, and audit records are additional. These are capacity assumptions; measure the deployed representation before allocating storage.
Count cardinality separately.
An event ID used as a metric label would introduce 200,000 distinct label values per day under that assumption, before other dimensions are combined. Keep event identifiers in controlled audit records. Prefer bounded metric dimensions such as outcome, event type, and processing stage; scrutinize tenant labels as the tenant count grows.
Sampling diagnostic traces trades debugging detail for a smaller footprint. However, required access evidence must remain reconstructable without sampled traces. Keep the committed authorization decision independently, and decide explicitly how much detail to retain about duplicate attempts. An aggregate duplicate count cannot explain a disputed access attempt.
The deduplication horizon is a correctness boundary, not just a storage setting. Fourteen days is appropriate only if the supported retry and replay window, including operational delays, fits inside it. Older manual replays require longer key retention or a controlled procedure that preserves original identity and checks existing business state. Deleting raw payloads and deleting event keys are separate decisions.
Rejected option: exhaustive logs and an in-memory set
An in-memory set plus full request logging makes the first implementation short. Its limitation is recovery: restarting discards keys, and logs provide evidence without enforcing uniqueness. Capturing full requests also expands exposure of credentials and shipment data.
A process-local cache remains useful as an optimization after the durable boundary exists. Eviction must never permit another committed mutation, and a cache hit must never acknowledge uncommitted work.
Before rollout, verify concurrency, crash recovery, authorization, and retention behavior with retries disabled. Assign ownership of reconciliation and replay permissions. Enable retries when the durable acceptance boundary and its access evidence survive those tests.
References
- OWASP Secrets Management Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html
- CloudEvents specification, including event identity attributes: https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/spec.md
- HTTP semantics, including idempotent methods and retry considerations: https://www.rfc-editor.org/rfc/rfc9110.html
Top comments (1)
Curious what you do when the same source, tenant, and event ID shows up again with a changed payload after the original shipment event committed. My dedupe rule keys only on those three, so the second insert fails and I log it as a correction that needs a new event ID. Otherwise a legit fix never lands. fwiw I built Webhook Relay, it keeps every request so you can replay them.