Short answer: use cron to discover which active SaaS customers need a weekly email digest, then put one idempotent background job per customer onto a queue; retain terminal job records briefly, but keep delivery evidence and unresolved retries for the full 7-day policy window.
The cheapest easy setup is not automatically “cron only.” The bill is made of trigger executions, queue operations, worker compute, database writes, payload bytes, and retained history. For a weekly digest, worker compute and message count usually track the number of active customers, while retention tracks both that volume and how long failures remain eligible for another attempt. Before selecting software, write the rough monthly model: active customers × weekly runs × attempts per digest, then add storage for the records you deliberately keep.
That choice also defines the delivery guarantee. A timer says when work becomes due. It doesn't prove that file processing finished, email sending was accepted, or a webhook sync reached its destination.
Timers don't know.
A durable queue plus an idempotency ledger gives those stages separate, inspectable outcomes — and separate retry decisions.
What are the real queue, cron, and retention costs?
Start with units, not product pricing. Let C be active customers, W the number of weekly runs in the billing period, A the mean processing attempts per digest, and D the mean number of days an unfinished record stays live. The dominant variable cost is commonly proportional to C × W × A; retained state is proportional to C × W × D. This isn't a benchmark. It is an accounting frame that exposes which measurement your own workload must supply.
| Cost term | What increases it | What changes it materially |
|---|---|---|
| Scheduled triggers | One global scan versus one timer per customer | Prefer one bounded discovery run when every customer shares the cadence |
| Queue operations | Publish, lease, acknowledge, and retry activity | Enqueue one digest workflow, not one message for every intermediate step |
| Worker compute | File generation, template rendering, and provider calls | Cache immutable inputs and stop retries after a terminal outcome |
| State retention | Attempt records, payloads, responses, and audit evidence | Store references and hashes instead of generated files when regeneration is safe |
| Delivery operations | Email attempts and webhook sync attempts | Retry only transient outcomes; quarantine permanent failures |
The change that often moves the dominant term is coarsening the job boundary. If a digest contains 40 report rows, forty tiny jobs create more leases, acknowledgements, and state transitions than one customer-level job. But making the job too large is risky: retrying a combined batch can repeat work for customers whose digests already succeeded. The useful boundary is one business side effect per independently retryable record. For this scenario, that is usually one customer digest, with file creation, email submission, and optional webhook sync represented as named stages in its state.
Don't optimize the wrong line item.
A single weekly cron invocation is cheap in conceptual and operational terms; the long tail comes from failed destinations held for 7 days, especially when every attempt preserves a generated attachment and verbose response body. Keep the job key, stage status, attempt count, next-attempt time, content hash, and compact provider reference. Drop reproducible files after success. The trade-off is forensic depth: if the original source data can change, a hash proves which content version was intended but cannot recreate bytes that were discarded.
How should a beginner SaaS combine cron, a background job queue, and retries?
Treat cron as a scanner. Once per week it reads a stable eligibility window, determines which customers were active, and inserts jobs using a uniqueness key such as weekly-digest:{customer_id}:{period_start}. A uniqueness constraint makes a repeated scan harmless. This matters because scheduled systems can run late, an operator can trigger a recovery run, and overlapping deployments can briefly start two schedulers.
Then let workers claim due records. A claim needs a lease deadline so another worker can recover work after a process exits before acknowledgement. RabbitMQ documents the same underlying delivery idea through consumer acknowledgements: acknowledgement tells the broker that delivery was processed, while an unacknowledged delivery can be made available again. That is at-least-once processing, so application-level idempotency still matters.
The state transition should be transactional: record the successful stage and its stable external reference before acknowledging the job. For an email stage, the idempotency key belongs to the digest, not the individual attempt. For webhook sync, send an event identifier that the receiving side can deduplicate. File processing should write to a deterministic object name or compare a content hash before replacing output. Those rules let a retry repeat computation without repeating the business effect.
Keep the retry schedule in data rather than sleeping inside a worker. A practical policy might calculate increasing delays, cap them, add random jitter, and stop at first_attempt_at + 7 days. The exact intervals depend on provider rate limits and the business promise; I'm not sure there is one defensible universal sequence. Evidence from your own transient-failure distribution would resolve that choice. The fixed requirement is simpler: every pending record has an explicit next_attempt_at, and the scanner or queue can recover it after a restart.
Cron alone can implement that database-backed pattern at small scale: select due rows, claim them atomically, process them, and update their state. It becomes a queue in all but name. A dedicated broker earns its place when work must start promptly, concurrency needs active control, or the backlog must drain independently of the web process. Keep cron-only when a delayed weekly run is acceptable, volume fits comfortably inside one bounded batch, and the team is prepared to operate the claims table correctly.
Make the ledger the source of delivery truth
Here is the core in Python, using a generic repository and sender rather than a product SDK. The transaction boundary is intentionally visible. A production repository also needs atomic lease acquisition and a uniqueness constraint on job_key; hiding those behind convenient method names does not remove the requirement.
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
@dataclass
class DigestJob:
job_key: str
customer_id: str
period_start: str
first_attempt_at: datetime
attempt_count: int
content_hash: str
def process_digest(job: DigestJob, repo, renderer, mailer, webhook) -> None:
if repo.is_terminal(job.job_key):
return
try:
artifact = renderer.render(
customer_id=job.customer_id,
period_start=job.period_start,
)
repo.assert_content_hash(job.job_key, artifact.sha256)
receipt = mailer.send(
idempotency_key=job.job_key,
recipient=repo.recipient_for(job.customer_id),
attachment=artifact.bytes,
)
repo.record_email_accepted(job.job_key, receipt.reference)
if webhook is not None:
webhook.send(
event_id=f"{job.job_key}:delivered",
payload={"customer_id": job.customer_id, "period": job.period_start},
)
repo.record_webhook_synced(job.job_key)
repo.mark_complete(job.job_key, completed_at=datetime.now(timezone.utc))
except repo.TransientFailure as error:
deadline = job.first_attempt_at + timedelta(days=7)
next_attempt = repo.retry_time(job.attempt_count + 1)
if next_attempt < deadline:
repo.reschedule(job.job_key, next_attempt, error.category)
else:
repo.mark_expired(job.job_key, error.category)
raise
except repo.PermanentFailure as error:
repo.mark_failed(job.job_key, error.category)
raise
The code distinguishes “accepted for sending” from inbox delivery. I've seen email and OTP flows lose that distinction under spam filtering, rate limits, and delivery gaps; a successful API call cannot establish that a person received a message. Capture later delivery events if the provider supplies them, but don't make the weekly worker wait for those asynchronous signals. Its durable result is the submission reference. A separate event consumer advances delivery status and deduplicates repeated callbacks.
There is another edge: a crash can occur after the external sender accepts the email but before record_email_accepted commits. The safest resolution is an idempotency key honored across attempts. If the external system cannot provide that contract, exactly-once sending is not available; your ledger can reduce duplicates but cannot prove they never occur.
Say this plainly.
Where does cron stop being the right tool?
Cron isn't a good fit when each customer needs a different send time, retries must wake promptly, a burst must respect downstream rate limits, or file processing can outlive the scheduler's execution window. A queue is the clearer control plane in those cases because pending work, concurrency, and acknowledgement are first-class concerns.
The catch is operational weight. A broker adds deployment, monitoring, dead-letter handling, and retention policy decisions. A beginner SaaS with a few hundred weekly digests may get a better failure boundary from a relational claims table plus one cron scan, provided claims are atomic and workers are restartable. GitHub Actions can supply a scheduled trigger for low-stakes automation, but its documentation warns that scheduled workflows may be delayed during high-load periods, especially near the start of an hour. That boundary makes it unsuitable as the sole timing guarantee for a customer-facing send deadline.
Measure lag from due_at to first claim, attempts per terminal job, oldest pending age, duplicate suppression, and time spent in each stage. Alert on the promise, not merely on worker health. A process can be green while a rate-limited destination has accumulated two days of undelivered digests.
Finally, delete with intent. After the 7-day retry window, stop keeping generated files, stack traces, and full response bodies unless compliance or dispute handling requires them. Retain the compact audit row for the period your policy demands: job key, intended recipient reference, content hash, timestamps, terminal state, and external reference. This lowers storage and exposure, but it costs diagnostic detail when a customer asks what a discarded attachment contained. If exact reconstruction is legally or operationally required, encrypted immutable retention is the better trade-off even though it makes storage the larger term.
Top comments (0)