DEV Community

MordecaiNilsson7582
MordecaiNilsson7582

Posted on

Marketplace Reconciliation: Long Background Jobs, Queue Workers, and Cron Timeouts

Short answer: make the cron trigger enqueue a small, idempotent reconciliation task, then let workers process checkpointed chunks instead of trying to finish the payment-provider scan inside one 15-minute execution.

That decision keeps the failure boundary visible. A scheduler starts a run. A queue delivers bounded work. A database records what was applied. The worker can stop after one chunk and continue from a durable cursor, which is much easier to test than a long loop that only knows it ran for 14 minutes.

This is the shape I use for notebook-to-prod transitions and eval-driven systems: the run identity, input revision, and progress marker are data, not assumptions hidden in process memory. Prompt-cost awareness matters here too. If reconciliation also creates an explanation or classification, store that output per payment chunk so a retry does not silently spend tokens twice.

How should a background job queue handle long-running worker tasks after a cron trigger?

Start with a stable run ID for the intended business interval, such as 2026-08-09, and split the provider result set into deterministic chunks. A chunk might contain payment IDs from a stored range or a page token captured in the run record. Do not make “the next ten minutes” the checkpoint: after a timeout, a retry cannot tell which side effects committed during that interval.

The worker should claim (run_id, chunk_id) transactionally, apply each payment update with the same idempotency key, and commit the result before it advances the cursor. If that key already exists, the operation becomes a no-op. Queue delivery can still happen twice; correctness belongs in the consumer and its storage layer.

Keep it boring.

A small Python model makes the state transition concrete:

import sqlite3


def open_database(path="reconciliation.sqlite"):
    database = sqlite3.connect(path)
    database.execute(
        "CREATE TABLE IF NOT EXISTS chunks ("
        "run_id TEXT NOT NULL, chunk_id INTEGER NOT NULL, "
        "status TEXT NOT NULL, PRIMARY KEY (run_id, chunk_id))"
    )
    database.execute(
        "CREATE TABLE IF NOT EXISTS reconciled_payments ("
        "run_id TEXT NOT NULL, payment_id TEXT NOT NULL, "
        "PRIMARY KEY (run_id, payment_id))"
    )
    return database


def reconcile_chunk(run_id, chunk_id, payment_ids, database_path):
    with open_database(database_path) as database:
        row = database.execute(
            "SELECT status FROM chunks WHERE run_id = ? AND chunk_id = ?",
            (run_id, chunk_id),
        ).fetchone()
        if row and row[0] == "done":
            return "already_done"

        database.execute(
            "INSERT OR REPLACE INTO chunks "
            "(run_id, chunk_id, status) VALUES (?, ?, 'done')",
            (run_id, chunk_id),
        )
        # Apply each provider result using (run_id, payment_id) as its key.
        # The real payment update belongs inside this transaction boundary.
        for payment_id in payment_ids:
            database.execute(
                "INSERT OR IGNORE INTO reconciled_payments "
                "(run_id, payment_id) VALUES (?, ?)",
                (run_id, payment_id),
            )
    return "processed"
Enter fullscreen mode Exit fullscreen mode

The example is intentionally an adapter-free state machine. A Node.js service can use the same fields and transitions; changing language should not change the retry contract. The example creates both tables for clarity, while a production migration, not the worker startup path, should own schema creation and make the composite keys explicit.

One detail deserves extra care: publishing the next chunk and committing the current one are separate effects. If the process dies between them, a run can be marked done with no follow-up message. A periodic reconciler can find a committed chunk whose successor is absent and publish that successor using the same run identity. It must use an outbox or an equivalent uniqueness constraint, so recovery itself is idempotent.

Where does the 15-minute limit belong in the architecture?

Put it around the scheduler invocation, not around the whole business operation. The cron handler validates its request, creates the run row, and enqueues chunk zero. It should return quickly. The worker receives one bounded message, records its outcome, and schedules or exposes the next chunk for delivery.

Measure chunks from real provider latency before setting their size. A payment page that is safe in a local notebook can become unsafe when the provider slows down, the database is locked, or an explanation model adds token latency. Leave headroom for acknowledgement and commit work. Shorter chunks increase queue traffic and bookkeeping; larger chunks increase duplicate work and recovery time. There is no universal best number.

The queue message should carry identifiers and a compact cursor, not the entire payment export. Keep the source snapshot in application storage and bind the cursor to its revision. This prevents a retry from reading a different dataset and reporting a result that cannot be reproduced. It also makes eval-driven testing practical: feed the same snapshot twice and assert that the second delivery changes no reconciliation record or model-cost record.

Which failure modes should the worker make boring?

At-least-once delivery means duplicate messages are normal. A unique database key is stronger than an in-memory set, because the latter disappears when the process restarts. A timeout is also normal: retry the unfinished chunk, not the entire nightly run.

The dangerous case is a remote side effect without a matching durable identity. For each payment, send or store an idempotency key that survives retries. If the provider supports such keys, use the provider's documented contract; if it does not, record the attempted operation and reconcile the provider's final state before retrying. Do not infer “paid” from a worker acknowledgement.

Consider a run with three chunks. Chunk zero updates the local ledger and commits, then the worker process exits before it publishes chunk one. A naive retry of the whole run may ask the provider about the same payments again, overwrite a manually corrected status, and charge for a second explanation request. A cursor-only design has the opposite problem: it may advance past a payment whose remote call completed but whose local commit did not. The durable identity solves neither uncertainty automatically; it gives the retry a place to ask the provider for final state and a unique key with which to record the answer. The recovery reconciler then sees that chunk zero is committed, sees that chunk one is missing, and publishes exactly one successor. If chunk one is delivered twice, both deliveries converge on the same (run_id, chunk_id) record. This is the useful level of detail to exercise in an eval harness: inject exits at each boundary, assert the final ledger, and separately count provider calls and model tokens.

The operational signals should answer three questions: which run is stuck, which chunk is being retried, and whether the provider state agrees with the local state. Log run ID, chunk ID, payment count, attempt count, elapsed time, and a redacted provider request ID. Keep secrets and payment details out of logs. A single alert on “queue depth is high” is not enough; a small queue with one poisoned chunk can be more urgent.

Failure signal Durable evidence Action
Worker timeout Chunk is still started Retry only that chunk after checking remote state
Duplicate delivery The composite idempotency key exists Return success without applying it again
Missing successor Current chunk is done, next message is absent Let the reconciler publish one successor
Provider disagreement Local and remote status differ Quarantine the payment for review

I treat an HTTP 409 from the application's idempotency layer as a successful duplicate, while an authentication failure or a provider rate-limit response needs a different retry policy. That distinction is a useful test case, not a reason to bury all errors under one catch-all retry. Your mileage may vary on backoff values because provider quotas and business deadlines differ, but the state transitions should stay fixed.

What should be checked before shipping the nightly run?

First, replay the same chunk and verify that payment state, output rows, and any token-cost record remain unchanged. Then kill the worker after the provider call but before the local commit, and repeat the delivery. Finally, kill it after the commit but before publishing the successor, and verify that the reconciler repairs the missing handoff without creating a second chunk.

The run needs a deadline and an operator path. Mark a run as incomplete when its business window closes, preserve the last successful cursor, and make a repair or resume action explicit. A poison payment should be isolated after a bounded number of attempts; otherwise one malformed record can consume the whole nightly budget. The repair queue should retain enough context to explain why that payment was deferred.

This pattern is not suitable when the job needs a visual DAG, a native fan-out/join graph, long-lived human approvals, or coordinated timers across many services. Choose a workflow orchestrator for those requirements. Stick with a plain queue when the work is a sequence of independently retryable chunks and the application already owns the state that makes retries safe.

The final checklist is short: freeze the input revision, assign a deterministic run ID, choose a chunk boundary, persist idempotency keys, leave timeout headroom, instrument every transition, and test crashes at both sides of each commit. The code is the easy part. The contract between the provider, database, and queue is what lets the job finish after the original trigger is long gone.

References

Top comments (0)