Marketplace Digests: Node.js Queue Retry, DLQ Redrive, API Calls, and Backlog Telemetry

Short answer: keep the weekly schedule cheap and deterministic by making it enqueue eligible customer IDs, then let a rate-aware worker own API calls, bounded retries, dead-letter routing, redrive, and backlog monitoring. For a marketplace digest, latency and cost pull in opposite directions: concurrency clears the queue sooner, while deliberate pacing avoids wasted attempts and keeps downstream pressure predictable.

A weekly digest is forgiving about seconds and unforgiving about duplicates. The scheduler should therefore create a run record and immutable per-customer jobs; it shouldn't render recommendations, call an email API, or wait for the whole campaign. The data flow is plain: select active customers, enqueue one job per recipient with a stable idempotency key, claim jobs under a shared rate budget, send, and move terminal failures into a dead-letter queue (DLQ). A separate redrive operation returns reviewed jobs to the live queue.

Keep those boundaries sharp.

Start with a testable storage contract

The worker below is deliberately storage-agnostic. Its repository methods stand for atomic database operations: claim_ready must lease only jobs whose available_at has passed, and each transition must verify the lease owner. That contract can sit behind a Node.js queue consumer in production, while the compact Python reference keeps the state transitions easy to exercise in a notebook and an eval harness.

from __future__ import annotations

import asyncio
import random
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from enum import Enum
from typing import Protocol


class ResultKind(str, Enum):
    SENT = 'sent'
    RATE_LIMITED = 'rate_limited'
    TRANSIENT = 'transient'
    PERMANENT = 'permanent'


@dataclass(frozen=True)
class Job:
    id: str
    campaign_id: str
    customer_id: str
    attempts: int

    @property
    def idempotency_key(self) -> str:
        return f'{self.campaign_id}:{self.customer_id}'


@dataclass(frozen=True)
class DeliveryResult:
    kind: ResultKind
    retry_after_seconds: int | None = None
    reason_code: str | None = None


class Repository(Protocol):
    async def claim_ready(self, limit: int, lease_seconds: int) -> list[Job]: ...
    async def mark_sent(self, job_id: str) -> None: ...
    async def retry_at(self, job_id: str, available_at: datetime, reason: str) -> None: ...
    async def move_to_dead(self, job_id: str, reason: str) -> None: ...


class DeliveryClient(Protocol):
    async def send_digest(self, customer_id: str, idempotency_key: str) -> DeliveryResult: ...


def retry_delay(attempt: int, retry_after_seconds: int | None) -> timedelta:
    if retry_after_seconds is not None and retry_after_seconds >= 0:
        return timedelta(seconds=min(retry_after_seconds, 3600))
    ceiling = min(15 * (2 ** attempt), 3600)
    return timedelta(seconds=random.uniform(0, ceiling))


async def process_job(
    repo: Repository, client: DeliveryClient, job: Job, max_attempts: int = 6
) -> None:
    result = await client.send_digest(job.customer_id, job.idempotency_key)
    if result.kind is ResultKind.SENT:
        await repo.mark_sent(job.id)
        return

    reason = result.reason_code or result.kind.value
    next_attempt = job.attempts + 1
    if result.kind is ResultKind.PERMANENT or next_attempt >= max_attempts:
        await repo.move_to_dead(job.id, reason)
        return

    delay = retry_delay(next_attempt, result.retry_after_seconds)
    await repo.retry_at(job.id, datetime.now(timezone.utc) + delay, reason)


async def drain_once(repo: Repository, client: DeliveryClient) -> None:
    jobs = await repo.claim_ready(limit=20, lease_seconds=90)
    semaphore = asyncio.Semaphore(5)

    async def guarded(job: Job) -> None:
        async with semaphore:
            await process_job(repo, client, job)

    await asyncio.gather(*(guarded(job) for job in jobs))

Those constants are policy inputs, not universal recommendations. The 20-job claim, 5 in-flight calls, 90-second lease, six-attempt ceiling, and one-hour backoff cap make the example concrete enough to test, but production values must come from the downstream quota, observed call latency, digest deadline, and business tolerance for delayed mail. Your mileage may vary, especially when one API account is shared by several workloads.

The test suite should use a fake clock and a scripted delivery client. Verify that a 429-like rate-limited result is scheduled in the future, a permanent result never retries, the sixth attempt becomes dead, two workers can't claim the same lease, and replaying a sent idempotency key doesn't produce a second logical delivery. Also test the awkward boundary where the lease expires during a slow call. These are evals for infrastructure behavior: fixed inputs, explicit state transitions, and assertions over both the job row and emitted metrics.

Notebook-to-prod is useful here. First explore delay distributions and state transitions with an in-memory repository; then run the identical tests against the transactional repository used by workers. The notebook is for understanding the policy. The database is the authority.

How should a Node.js queue retry failed rate-limited API calls?

Treat delivery as a state machine rather than a loop around an API client. A job starts as queued, becomes leased while one worker owns it, and ends as sent, retry_wait, or dead. Only a confirmed delivery becomes sent. A rate-limited response belongs in retry_wait with a future availability time; a permanent failure or a job that exhausts its attempt budget belongs in dead. That distinction prevents a bad address from consuming the same capacity as temporary throttling.

The retry delay should respect a valid server-provided delay when the downstream API supplies one. Otherwise, use capped exponential backoff with jitter. The cap matters because a campaign delayed by hours shouldn't create absurd multi-day retry times, and jitter matters because thousands of jobs becoming eligible on the same second can recreate the overload. Don't retry every failure: authentication and invalid-recipient failures require correction, not patience. Keep the classification table close to the adapter for the downstream API, since response semantics vary.

For the weekly marketplace run, use two independent controls. A token bucket or equivalent limiter controls request starts per time window. A smaller concurrency limit controls simultaneous in-flight requests. They answer different questions. The rate limit protects the provider quota; concurrency controls sockets, memory, and how many ambiguous outcomes exist when a worker stops. Increasing concurrency above the allowed request rate can raise cost through failed attempts without improving useful throughput.

Idempotency closes the most dangerous gap — the worker may lose its lease after the downstream service accepts a message but before local state records success. Build a stable key from the campaign and customer, such as weekly-2026-W32:customer-1842, and pass it through whenever the delivery API supports idempotency. Locally, enforce uniqueness on that key. If the remote API doesn't provide an idempotency contract, the honest limitation is that exactly-once delivery can't be guaranteed across the network; choose an application policy that tolerates a rare duplicate or add a reconciliation step based on provider delivery records.

One key. One logical send.

Treat the DLQ as a forensic dataset

A dead-letter queue is evidence, not a trash can. Store the original payload or a durable reference, campaign ID, customer ID, attempt count, first and last failure timestamps, a normalized reason code, and the worker version that made the decision. Avoid secrets in payloads and logs. API credentials should be loaded through the deployment's secret-management path, scoped to the required operation, rotated according to policy, and never copied into a job body; the OWASP key-management guidance is a useful baseline for the lifecycle around those keys.

Redrive should be explicit and auditable. Select jobs by a narrow predicate such as campaign plus normalized reason, record a redrive batch ID and operator or automation identity, reset only the fields required by policy, and enqueue them at a controlled rate. Preserve prior attempts and failure history. If 18,000 jobs failed because recipient data was invalid, putting all 18,000 straight back into the live queue after a data fix can starve current work and hit the same downstream limit again. Canary a small batch, compare its terminal outcomes with the pre-redrive reason, and expand only when the signal is clean.

Never make redrive mean infinite retry.

The catch is that a DLQ adds operational surface: retention rules, sensitive-data review, access control, tooling, and an ownership rota. It isn't suitable as the primary mechanism for expected long delays or routine quota smoothing; use scheduled availability in the live queue for those. Stick with a simpler retry table when volume is low, one team owns the entire path, and the database can provide atomic claims without turning cleanup into a second platform. Choose a dedicated broker when multiple consumers, high sustained throughput, or independent retention and replay controls justify the extra moving parts.

Let the campaign deadline set the worker budget

Queue depth alone is ambiguous. Ten thousand jobs can be healthy early in a weekly campaign and alarming near its deadline. The leading service-level signal is oldest eligible job age, split by campaign and priority. Pair it with ready depth, retry-wait depth, lease count, completion rate, dead-letter rate, attempt distribution, and downstream outcome classes. For cost, track API attempts per successful digest and rendering work per terminal delivery; retries that never create value are the first place latency tuning becomes a spend problem.

A practical forecast uses remaining eligible jobs divided by the recent successful completion rate, with a safety margin for retries. Compare the projected drain time with the campaign deadline. If the estimate crosses the deadline, first confirm the bottleneck: a rate budget, concurrency, slow rendering, or a growing permanent-failure class. Raising worker count helps only some of those. If the downstream limit is binding, more workers mostly compete for the same tokens.

Alert on trends that demand an action. Oldest-job age approaching the digest deadline deserves a page; a small DLQ increase with one stable reason may deserve a ticket; zero throughput while ready jobs exist deserves immediate investigation. Dashboards should expose each retry reason separately, because a combined failed counter hides whether the remedy is waiting, correcting customer data, rotating credentials, or pausing a campaign. I'm not sure a universal alert percentage is defensible without the campaign window and normal traffic curve. Establish thresholds from load tests and several observed runs, then reevaluate them when the customer population or downstream contract changes.

Deployment follows the same logic. Ship the scheduler, worker, and redrive command as separately controllable units. Before enabling a campaign, verify the unique idempotency constraint, lease recovery, credential scope, rate budget, retry classifier, DLQ retention, and dashboard dimensions in a staging run with a fake delivery client. Roll out with a small customer cohort, inspect completion and attempts-per-success, then increase the cohort. The weekly scheduler remains a thin trigger throughout; hosted cron documentation is useful for understanding trigger behavior, but durable campaign progress belongs in application state rather than in one scheduler invocation.

The final decision rule is compact.

Favor lower concurrency and predictable cost when the projected drain time remains inside the digest deadline; spend additional concurrency only when measurements show local capacity, rather than the downstream API limit, is the constraint. Keep dead letters reviewable, redrive them in canaries, and make oldest-job age the clock everyone watches.

References

• https://vercel.com/docs/cron-jobs
• https://cheatsheetseries.owasp.org/cheatsheets/Key_Management_Cheat_Sheet.html