DEV Community

matsjohansson6547
matsjohansson6547

Posted on

B2B SaaS Reconciliation Scheduling: Recoverable Email, SMS, and Push Reminders

Short answer: for nightly payment reconciliation, use cron only to release durable jobs, use a queue to isolate each account's work, and store reminder delivery state separately from both; this is the simplest design that lets an operator retry email, SMS, or push without rerunning a payment decision.

The deciding feature isn't a pleasant scheduling API. It's operational recovery. A B2B SaaS system can survive a late nightly run, but it must not turn one uncertain payment-provider response into duplicate customer reminders or force an engineer to reconstruct intent from logs. The useful boundary is small: the scheduler says when, the queue controls work, and a database ledger says what happened.

That split also fits an AI-heavy Python service. Reconciliation output may eventually feed a classifier or generate reminder copy, but deterministic state should decide whether any message is eligible. Keep prompts out of the control plane. An eval can judge copy quality later; it cannot repair a missing idempotency key.

Clock first.

How can a SaaS scheduling API recover user reminders across email, SMS, and push?

Model the workflow as two linked state machines. The first owns reconciliation for one tenant and one billing window. The second owns one delivery attempt for one channel. A successful reconciliation may create zero or more reminder records, but a reminder worker never calls the payment provider and a reconciliation retry never sends a message directly.

In plain language, a nightly cron tick inserts due reconciliation jobs. Workers claim those jobs from a queue, read the payment provider, and commit the result. That same database transaction records any email, SMS, or push intents. Separate delivery workers then claim those intents. Every durable record carries a stable idempotency key, an attempt count, and a lease deadline so abandoned work becomes eligible again.

This answers the cron-versus-message-queue question without pretending they are interchangeable. Cron is a clock. A queue is a pressure and retry boundary. The ledger is the evidence needed when an operator asks, "Did account acct_1042 receive the overdue notice, or did a worker merely start it?"

Use a scheduling API only if it preserves that evidence or lets your application preserve it. An API that accepts run_at but cannot expose stable job identity, cancellation state, attempt history, and redrive behavior is simple during setup and expensive during recovery. The catch is that a full queue is unnecessary for a tiny internal task where occasional manual reruns are harmless. In that case, cron plus a unique database row can be the clearer choice.

Testing the Python recovery slice

The following standard-library example is intentionally narrow. It creates one reconciliation job, claims it with a lease, commits three channel-specific reminder intents, and delivers each intent with a deterministic key. SQLite stands in for any transactional database; print stands in for a channel adapter. The architecture matters more than the adapter.

from __future__ import annotations

import json
import sqlite3
import uuid
from datetime import datetime, timedelta, timezone


def utc_now() -> datetime:
    return datetime.now(timezone.utc)


def timestamp(value: datetime) -> str:
    return value.isoformat(timespec="seconds")


db = sqlite3.connect(":memory:", isolation_level=None)
db.row_factory = sqlite3.Row
db.executescript(
    """
    CREATE TABLE reconciliation_jobs (
        id TEXT PRIMARY KEY,
        account_id TEXT NOT NULL,
        billing_date TEXT NOT NULL,
        due_at TEXT NOT NULL,
        status TEXT NOT NULL,
        attempts INTEGER NOT NULL DEFAULT 0,
        lease_until TEXT,
        UNIQUE (account_id, billing_date)
    );

    CREATE TABLE reminder_deliveries (
        id TEXT PRIMARY KEY,
        reconciliation_id TEXT NOT NULL,
        channel TEXT NOT NULL,
        recipient TEXT NOT NULL,
        payload TEXT NOT NULL,
        status TEXT NOT NULL,
        attempts INTEGER NOT NULL DEFAULT 0,
        lease_until TEXT,
        idempotency_key TEXT NOT NULL UNIQUE,
        UNIQUE (reconciliation_id, channel, recipient)
    );
    """
)


def enqueue_reconciliation(account_id: str, billing_date: str) -> None:
    job_id = str(uuid.uuid5(uuid.NAMESPACE_URL, f"{account_id}:{billing_date}"))
    db.execute(
        """
        INSERT OR IGNORE INTO reconciliation_jobs
            (id, account_id, billing_date, due_at, status)
        VALUES (?, ?, ?, ?, 'ready')
        """,
        (job_id, account_id, billing_date, timestamp(utc_now())),
    )


def claim_one(table: str, lease_seconds: int = 90) -> sqlite3.Row | None:
    allowed_tables = {"reconciliation_jobs", "reminder_deliveries"}
    if table not in allowed_tables:
        raise ValueError("unexpected table")

    now = timestamp(utc_now())
    lease_until = timestamp(utc_now() + timedelta(seconds=lease_seconds))
    db.execute("BEGIN IMMEDIATE")
    try:
        row = db.execute(
            f"""
            SELECT * FROM {table}
            WHERE status = 'ready'
               OR (status = 'working' AND lease_until < ?)
            ORDER BY id
            LIMIT 1
            """,
            (now,),
        ).fetchone()
        if row is None:
            db.execute("COMMIT")
            return None
        db.execute(
            f"""
            UPDATE {table}
            SET status = 'working', attempts = attempts + 1, lease_until = ?
            WHERE id = ?
            """,
            (lease_until, row["id"]),
        )
        claimed = db.execute(
            f"SELECT * FROM {table} WHERE id = ?", (row["id"],)
        ).fetchone()
        db.execute("COMMIT")
        return claimed
    except Exception:
        db.execute("ROLLBACK")
        raise


def finish_reconciliation(job: sqlite3.Row) -> None:
    recipients = {
        "email": "billing@example.test",
        "sms": "+15550101042",
        "push": "device_1042",
    }
    db.execute("BEGIN IMMEDIATE")
    try:
        for channel, recipient in recipients.items():
            delivery_key = f"{job['id']}:{channel}:{recipient}"
            db.execute(
                """
                INSERT OR IGNORE INTO reminder_deliveries
                    (id, reconciliation_id, channel, recipient, payload,
                     status, idempotency_key)
                VALUES (?, ?, ?, ?, ?, 'ready', ?)
                """,
                (
                    str(uuid.uuid5(uuid.NAMESPACE_URL, delivery_key)),
                    job["id"],
                    channel,
                    recipient,
                    json.dumps({"template": "payment_review", "account": "acct_1042"}),
                    delivery_key,
                ),
            )
        db.execute(
            """
            UPDATE reconciliation_jobs
            SET status = 'done', lease_until = NULL
            WHERE id = ?
            """,
            (job["id"],),
        )
        db.execute("COMMIT")
    except Exception:
        db.execute("ROLLBACK")
        raise


def deliver(reminder: sqlite3.Row) -> None:
    payload = json.loads(reminder["payload"])
    print(
        f"send {reminder['channel']} to {reminder['recipient']} "
        f"with key {reminder['idempotency_key']} and template {payload['template']}"
    )
    db.execute(
        """
        UPDATE reminder_deliveries
        SET status = 'sent', lease_until = NULL
        WHERE id = ?
        """,
        (reminder["id"],),
    )


enqueue_reconciliation("acct_1042", "billing-window-17")
reconciliation = claim_one("reconciliation_jobs")
if reconciliation:
    finish_reconciliation(reconciliation)

while reminder := claim_one("reminder_deliveries"):
    deliver(reminder)
Enter fullscreen mode Exit fullscreen mode

There is one deliberately exposed boundary: a process can stop after an external channel accepts a message but before the local sent update commits. The stable idempotency_key lets a channel adapter ask for deduplication where that capability exists. Where it doesn't, the system cannot promise exactly-once delivery. Don't hide that fact behind a retry decorator. Record the provider's receipt, make the uncertainty visible, and give operations a way to inspect before redriving. The UNIQUE constraints do a different job. They stop a repeated cron tick from creating a second reconciliation for the same account and billing window, and they stop a repeated reconciliation commit from multiplying delivery intents. Those constraints are the bridge from a notebook-shaped happy path to production behavior: rerunning the cell, handler, or worker should converge on the same durable records. For a production queue, claiming and lease renewal must match the database and queue semantics you actually deploy. The sample uses a short lease to keep the mechanism visible, not to prescribe a timeout. A payment-provider call, an email adapter, and an AI copy step can have very different latency distributions — especially when one path waits on generated copy — so measure them independently. I'm not sure what lease is right for your workload until those distributions and the provider timeout contract are known.

Retry and failure semantics matter more than scheduler syntax

Start by naming each failure boundary. If the cron tick runs twice, the reconciliation uniqueness key absorbs the duplicate. If a worker stops while holding a job, the lease makes the job claimable after expiry. If reconciliation succeeds but delivery workers are unavailable, committed reminder intents wait without querying the payment provider again. If one channel needs another attempt, only that channel's record moves; email success doesn't block an SMS retry.

This is where a dead-letter queue helps. It isolates messages that repeatedly fail processing so they can be inspected or redriven rather than cycling through the main workload. The AWS SQS documentation is useful primary evidence for the pattern even if the implementation uses another queue. It also warns that moving a message can affect ordering, which matters if reminders for the same account must remain sequential. A dead-letter queue is an operational lane, not a substitute for domain state.

Keep the retry policy boring. Retry only operations whose outcome is safe to repeat or protected by a stable key. Put attempt count, next eligibility time, and the last categorized outcome in durable state. A transport timeout means "unknown," not "failed"; treating those words as synonyms is how duplicates escape. Rate limiting and invalid recipient data need different actions, even though both can arrive through the same worker interface.

Unknown means stop.

Prompt-cost awareness belongs here too. If an AI step drafts channel copy, generate it once per content version and store the approved result before fan-out. Retrying a delivery should not spend tokens again or silently change the message. An eval harness can compare tone, policy compliance, and required payment fields on that stored candidate. The queue should move accepted artifacts, not improvise them.

Fast recovery can still be the wrong optimization. If legal or billing policy requires a human approval before every customer contact, automatic redrive is not suitable; route uncertain reminders to review. If the workload has strict per-account ordering, stick with a partitioned or ordered queue instead of a generic pool, and accept lower parallelism for that key. If reminders are best-effort internal nudges with low volume, a transactional table scanned by cron may be enough. More machinery creates more states for the team to operate.

Data governance defines the smallest honest architecture

Evaluate candidates with a forced recovery exercise, not a feature checklist. Seed two accounts, make one reconciliation worker lose its lease, repeat the cron tick, and pause one channel worker after external acceptance. Then ask whether the stored state tells an operator exactly which action is safe. This is an eval for infrastructure: fixed inputs, explicit invariants, and evidence that can be checked without reading a wall of logs.

The core invariants are compact. One account and billing window produce at most one reconciliation record. One reconciliation, channel, and recipient produce at most one reminder intent. A worker owns a job only for the lease interval. A retry preserves the same logical identity. Reconciliation completion and reminder creation commit together. Delivery evidence remains channel-specific.

Evidence survives workers.

Now compare implementation shapes against those invariants:

Shape Good fit Operational limitation
Cron plus a transactional table Low-volume jobs, one team, simple polling Application owns leasing, retry timing, and inspection tools
Cron releasing queue messages Parallel tenant work and isolated retries Requires both durable domain state and queue operations
Per-reminder scheduling API Many independent future timestamps Can scatter recovery evidence unless stable IDs map back to the ledger
Ordered queue partitions Strict sequencing per account Hot accounts can constrain throughput for their partition

Short APIs are pleasant, but a short call is not the same as a small operating model. Count the concepts an on-call engineer must understand at 02:00: job identity, lease ownership, retry eligibility, channel receipt, and redrive. A design that makes those concepts queryable usually beats one that compresses them into a single opaque schedule() call.

The implementation checklist is prose because the handoffs matter more than isolated boxes. Before deployment, define the reconciliation key and the delivery key, then prove duplicate insertion is harmless. Exercise lease expiry with a stopped worker. Verify that delivery retry never calls reconciliation and that copy generation never runs during transport retry. Put queue age, oldest ready job, attempts by outcome, and uncertain delivery count on the same operational view. Document who may redrive, which states require review, and how long evidence is retained. Finally, run the recovery exercise in staging whenever queue, database, payment-provider, or channel-adapter behavior changes.

Do that, and the choice becomes less mysterious. Cron can wake the system, a message queue can distribute pressure, and a ledger can make the result recoverable. None of the three should impersonate the others.

References

Top comments (0)