Most apps with background jobs start the same way: the web app and the workers live in the same codebase, built into the same image, deployed together. The worker is just the app started with a different command.
At some point, someone suggests splitting them: separate deploys, maybe separate services, maybe separate teams. The argument that follows is one of the more interesting ones in backend architecture, because both sides are right about different things.
Here's my attempt at a fair summary. Then tell me where you land.
The case for deploying together
One version of the truth. When the web app enqueues a job, the worker that picks it up runs the same code. The job's payload format, the database schema and the business logic all match. A whole class of bugs, "the worker didn't understand the new job format", mostly disappears.
Simpler pipelines. One build, one artifact, one deploy. Rollbacks are one action. Fewer moving parts to monitor.
Shared code without ceremony. Models, validation and helpers are just imports. No shared library to version and publish.
Easier local development. One repo, one setup, everything runs together.
The case for deploying separately
Different reliability needs. A web request takes milliseconds; a job might run for twenty minutes. Restarting workers on every web deploy means interrupting long jobs constantly, or making every deploy wait for them to finish.
Different scaling. Web traffic and job volume rarely peak at the same time. Separate deployments make it natural to scale them independently.
Smaller blast radius. A bad change to a report-generation job shouldn't require redeploying the checkout page, and a web deploy shouldn't risk the payment retry worker.
Release cadence. Web code often ships many times a day. Workers that process money, emails or data imports may deserve a more careful pace.
The part both camps underestimate: jobs in flight
Whichever you choose, there's a moment that matters more than the architecture diagram: the deploy, with jobs still in the queue.
Consider a job enqueued by the old code and processed by the new code, or vice versa. This happens in both setups:
- Deploying together, rolling deploys mean old and new versions run side by side for a while. A job enqueued by a new web instance can be picked up by an old worker.
- Deploying separately, the gap is longer and more explicit. The web app might be days ahead of the workers.
And for scheduled jobs, the gap can be enormous. A job scheduled today to run in 30 days will be executed by whatever code is deployed in 30 days. Its payload was written by today's code.
So the real requirement, in both camps, is the same:
- Job payloads must be backwards and forwards compatible, at least for a while. Add fields; don't rename or remove them in the same release that starts using the new ones.
- Version your payloads when changes are unavoidable, and make workers handle every version that might still be in the queue.
- Long-delayed jobs should carry minimal data: an ID to look up, rather than a snapshot of state that may be stale or in an old format by the time it runs.
{ "type": "send_renewal_reminder", "version": 2, "subscription_id": "sub_123" }
Deploying together reduces the window. It doesn't remove it.
Graceful worker restarts matter more than the topology
The other underestimated piece: what happens to a job that's running when a worker is told to stop? Workers need to stop taking new jobs, finish or safely abandon the current one within a timeout, and leave anything unfinished to be retried. If that isn't in place, deploy frequency, together or separate, translates directly into interrupted jobs.
What about database migrations?
Migrations sharpen the whole debate. If the web app and workers share a database, a schema change has to work with every version of both that might be running at once. The usual answer, expand first and contract later, applies in both camps: add the new column, deploy code that writes both, backfill, switch reads, and only then remove the old column in a later release. Separate deploys make that sequence longer, but also more visible, which some teams find is an advantage rather than a cost.
Where I land
For most small and medium teams: same repo, same image, separately scaled and separately restarted processes. Build once, deploy the same artifact to both, but don't force every web deploy to bounce long-running workers at the same moment. Treat job payloads as a public contract regardless.
Split into truly separate services when there's a real reason: different teams, very different scaling, or reliability requirements that justify their own release process. "It feels cleaner" isn't quite enough.
But I can see the opposite case clearly. If your workers handle money and your web app ships twenty times a day, a slower, separate release process for workers isn't bureaucracy. It's caution in the right place.
Questions to settle it for your team
- Do any jobs run longer than your typical deploy takes?
- Are job payloads versioned, and how long can a job sit in the queue?
- Do web and job load peak together?
- Who gets paged when a worker deploy goes wrong, and is it the same people?
Long-delayed jobs, whose payloads outlive several deploys, are the reason we think so much about this at delay.run: the code that schedules a job and the code that runs it are rarely the same version.
Related reading
- What happens to the cron job that's running when you deploy? Graceful shutdown for scheduled work
- Your background jobs need SLOs too. 'It eventually ran' isn't good enough
- Make scheduled job handlers idempotent: safe to run twice
- The transactional outbox, explained: how to save to the database and enqueue a job without losing either
So: same deploy or separate? What does your team do today, and what pushed you there? Make your case in the comments.
Top comments (9)
Even deployed together there's a window where old workers are still draining jobs enqueued by the new web version, or the reverse, so the payload format question doesn't fully disappear. Versioning the job payload and keeping workers able to read the previous version for one release covers both setups. Splitting deploys then becomes a scaling and blast radius question rather than a correctness one. Where do you land on a job enqueued by the new version that gets picked up after a rollback?
Agreed, the mixed-version window exists either way, so payload versioning isn't really optional. On a rollback I'd want old workers to either read the new payload (if the change was additive) or park those jobs in a holding queue until a fixed version rolls forward, rather than failing them and burning retries. That's a big part of why I like keeping payload changes additive for a release or two.
Parking jobs in a holding queue on rollback is a nice detail, since burned retries are the part that does real damage. How do you decide when a parked job is safe to release: a version tag on the payload, or a manual flip once the fix is out?
A version tag on the payload, mostly. Each parked job records the minimum worker version that can handle it, and a release that meets that bar drains the queue on its own, so nobody has to remember the manual flip. I'd still keep a manual release as a fallback for the odd case where the fix ships under a different shape.
A minimum worker version on the payload is neat, since the release that meets the bar drains the queue by itself and nobody has to remember the flip.
What happens on a rollback: does a job that a newer worker already picked up and half processed get parked again, or do you only park before first pickup?
On Kubernetes, the "finish or safely abandon within a timeout" part already has a number, and plenty of worker Deployments never change it. terminationGracePeriodSeconds defaults to 30 seconds. After SIGTERM the kubelet waits that long and then kills whatever is still running. So a 20-minute job on a default worker gets cut off half a minute into every rollout.
Quick way to see it across a cluster:
A worker showing 30 is running on the default. A preStop hook runs inside that same window, so it doesn't buy extra time. And raising the number only helps if the worker stops pulling new jobs as soon as SIGTERM arrives, which ties straight back to your point about graceful restarts.
Good catch, that 30-second default is exactly the kind of number nobody looks at until a long job dies halfway through a rollout. Handy one-liner too. For anything that can run longer than a grace period you'd realistically set, I'd rather have the job checkpoint and resume than keep raising the timeout.
함께 배포해도 큐 안에는 서로 다른 버전의 작업이 남는다는 지적이 중요하네요. 장기 예약에서 ID만 전달하는 안도, 실행 시점의 최신 값을 읽어야 하는 항목과 예약 당시의 의도를 유지해야 하는 항목을 나눠보면 좋겠습니다. 배포 토폴로지와 별도로 이 정책을 테스트해두면 “형식은 호환되지만 다른 내용으로 실행된 작업”도 점검할 수 있겠어요.
좋은 포인트예요. 실행 시점의 최신 값을 읽어야 하는 필드와 예약 당시의 의도를 그대로 지켜야 하는 필드를 나누는 건 페이로드 설계에서 꼭 필요한 구분 같아요. 예를 들어 이메일 주소는 최신 값을 읽고, 금액이나 할인율은 예약할 때 값을 고정하는 식으로요.