One morning, our daily automated code review flagged a refactor we had just pushed. The finding fit in one line: in the new code, nothing renews a running generation's heartbeat while the model is thinking, so a generation can be reclaimed as dead while it's still working, and the model call is paid for, then thrown away.
It was right, and checking it turned up a hole older than the refactor.
Some context. I run backend engineering at GoodBarber, an app platform, and our AI Extension Builder generates the code of an app section from a description, then edits it on request. Each generation runs as a job on a worker. This post is about how that worker tells the rest of the system it's alive, and why reasoning models broke the way it did.
How a generation says it's alive
A running generation holds a key in Redis that says "I'm alive", with a 90-second expiry. Whoever runs the job is supposed to renew it. The studio, when it polls a generation or when the user reloads the page, checks two things: is the key gone, and is the row older than 180 seconds? If both, the row is reclaimed and marked as failed. That's how we clean up after a worker that crashed mid-job.
The question is what counts as "renewing". In the worker, the key was refreshed when something happened: a token arrived from the model, or the pipeline moved to its next step. Activity meant the job was alive. During a model call that hadn't produced its first token yet, nothing happened, and silence meant it wasn't.
The refactor
We had finished a cleanup the day before and pushed it that morning: every generation would run through the worker, one execution path, and the older paths that ran inside the web request were deleted. About 2,250 lines went away.
One of those paths was the resume after approval. Some generations pause to ask the user for something, a permission or a choice, and resume once the user answers. That resume used to run inside the request, under a small context manager that started a thread renewing the key every 30 seconds, tokens or not. The code around it explained why: if the first token takes more than 90 seconds, the key expires and a live generation gets killed.
The refactor moved the resume onto the worker, which renewed on activity only. The call to the keepalive, and the comment that explained it, were in the code that got deleted.
Why it mattered that week
A reasoning model thinks before it writes. That thinking happens before the first token, and it grows with the reasoning effort you ask for. The day we finished the cleanup, we had also raised the reasoning effort of edits from low to medium and deployed it; edits got about 60% slower. A model that thinks for more than 90 seconds before its first token is, for an activity-driven heartbeat, a model that died.
What happens next depends on who looks first:
- The studio polls, finds the key gone and the row old enough, and marks the generation failed. The worker finishes anyway, and its result is refused. The call is paid for and thrown away, and the user sees an error.
- The stream to the browser decides the worker is dead and starts a second job. The call is paid for twice.
Resumed generations were the most exposed, because their row is created when the plan is made, before the user approves. By the time the job resumes, the row is often already past 180 seconds. The only thing left between it and the reaper is the key.
The older hole
The regression itself never reached production: the review flagged it the morning the refactor was pushed, and the fix shipped with the refactor the next day. But the activity-driven refresh wasn't new. Normal generations on the worker already worked that way. They were less exposed, because a fresh row takes longer to cross 180 seconds. Less exposed is not safe.
We had a failure state for exactly this, a row reclaimed while pending, and it was not empty.
The fix
One change in the worker: every job runs under the same keepalive the old path used, a thread that renews the key every 30 seconds whether tokens arrive or not. The resume, normal generations, edits and retries all go through it. Simplified:
def run_job(generation_id, job):
with keep_alive(generation_id, every_seconds=30):
return job.run()
The test makes the worker wait without receiving a single token, and checks that the key was renewed during the wait. Without the fix, it fails with the most honest assertion message I've seen in a while: "0 not greater than 0".
What the numbers say so far
We compared production generations across three periods, read-only, split by deploys: two weeks before the deploy that raised the effort; the two days after it, with the higher reasoning effort already live but neither the refactor nor the fix; and the four and a half days after the next deploy, which shipped the refactor and the fix together. Rates are shares of the generations started by real users in each period, and median times are compared with the first period.
| before | higher effort, no fix | after the fix | |
|---|---|---|---|
| failure rate | 1.2% | 1.6% | 0.6% |
| reclaimed while pending | 0.6% | 0.3% | under 0.1% |
| left pending after the plan | 0.5% | 0.2% | 0.6% |
| median time, new app | baseline | -2% | +7% |
| median time, edit | baseline | +62% | +56% |
The two later windows last only days, so read their columns as a direction, not a measurement.
Reclaims are what a missing heartbeat produces, whether the worker died or only looked dead, and they nearly disappeared: from 0.6% of generations in the two weeks before to under 0.1% after the fix. The failure rate nearly halved. So far, a fix written for a regression that never shipped also closed the main failure we already had.
One line didn't move, and it belongs in the table anyway. A row left pending is one where the plan was made, nothing was written to the row afterwards, and nobody came back to the page to trigger a reclaim: 0.5% of generations in the two weeks before, 0.6% in the four and a half days after, a rate that didn't improve. Whether its code phase never started or died without a trace, the rows don't say. That isn't the heartbeat. It's the next thing to look at.
The middle column matters too: with the higher effort live for two days, the rate of reclaims didn't go up, 0.3% against 0.6% before. The effort increase alone didn't show up there. It shows up in the last line: at the median, edits took about 60% longer.
Every silence is a budget
The heartbeat is the obvious one. It's not the only thing in a typical stack that reads silence as failure:
- the reaper's age threshold and the key's expiry;
- the stream to the client, if it gives up after a period with no event;
-
the reverse proxy: nginx's
proxy_read_timeoutdefaults to 60 seconds; - the CDN or load balancer in front: Cloudflare returns a 524 when the origin sends nothing for 125 seconds, its current default;
- the worker process timeout, if your app server kills requests that run too long.
When the model started writing within seconds, the silent stretch at the start of a call was short. A reasoning model can stay silent much longer: at medium effort, a first token after 90 seconds was plausible enough for the review to flag it, as a risk rather than a measurement. Each of those timeouts is a budget against thinking time, and raising the reasoning effort spends it.
What I'd keep
- Liveness must not depend on output. A heartbeat comes from a timer. Progress comes from tokens. They are two signals, and the second can stay silent for a long time.
- Raising the reasoning effort is an infrastructure change. List every timeout between the model and the user, and check it against the slowest first token you can expect.
- When you consolidate paths, read what each one was protecting. Our old path carried its guard and the reason for it. The diff that deleted it was green.
- Count "paid and thrown away" as its own failure, not as one more error in the failure rate.
Which timeout in your stack would a model thinking for two minutes hit first?
Top comments (6)
This is a clean example of conflating two signals that look identical until a reasoning model pulls them apart: "the process is alive" and "the process is making progress." Renewing the heartbeat on activity quietly assumes those are the same thing, and they were — right up until the first-token latency grew long enough that a healthy job and a crashed one became indistinguishable to the reaper.
We settled on exactly the separation your old resume path had by accident: a heartbeat thread that renews on a wall-clock timer independent of token flow, and progress tracked as its own metric that's allowed to stall without killing the job. The thing I'd flag for anyone reading: the bug only surfaced because your automated review caught a missing renewal in the diff — the 90s/180s window meant it was invisible under normal latency and would've waited for a slow generation to delete it in prod. Did you also add an alert for "reclaimed while a model call was outstanding," so the next regression shows up as a signal instead of a silently re-paid generation?
Not yet. The reclaimed row gets its own error tag, and that's what the table counts, after the fact. The better signal is on the worker's side: when it finishes and finds its row already reclaimed, it logs one line and drops its result. The reaper can't tell a dead worker from a silent one, but a worker that comes back to a reclaimed row was alive all along. Today that line is an info log and nothing alerts on it. It should be a metric.
One correction: the old path didn't have the timer by accident. It came with a comment saying why, and the refactor deleted both.
The branch I'd worry about is the stream deciding the worker is dead and starting a second job. The keepalive thread fixes the false death, but it also means a truly wedged model call now looks alive forever, because the thread renews whether or not the call is making progress. I'd want a hard ceiling on the call itself, separate from the heartbeat, so liveness and progress are two different signals.
For the paid-twice case, giving each attempt a generation number and only accepting the result from the current one makes the late finisher harmless even when the reaper guesses wrong. Did you see any rows where both jobs finished and the older result landed?
This is a good example of a bug that a cleanup creates rather than introduces. The keepalive thread and the comment explaining why it existed were both in the 2,250 lines that went away, and the remaining path renewed on activity only. On the same day, reasoning effort went from low to medium, so the silence before the first token got longer at exactly the wrong moment.
The comment from Pratik is the one I would act on. A thread that renews every 30 seconds fixes the false death, but it also means a truly wedged call looks alive forever. Two signals, one for "the process exists" and one for "the call is still within its budget", keep both failure modes visible.
At Ascendra Ventures we write this kind of thing down as an acceptance case on any job that waits on a model: hold the first token for longer than the lease and confirm the job survives, then hang it past the ceiling and confirm it fails cleanly. It is cheap to run and catches exactly this regression.
Are you planning to promote that "worker found its row already reclaimed" log line to a metric?
In our Python agent pipelines, the earliest failure was the HTTP client socket read timeout on streaming requests, before Redis or the queue reaper ever noticed.
Client libraries like httpx default read_timeout to 60 seconds and count time-to-first-token against that clock. When a reasoning model spends 90 seconds in silent thinking before emitting chunk zero, the client raises ReadTimeout and severs the TCP connection. The provider cluster keeps churning through the completion and bills for it anyway, while the worker catches the error and retries, immediately firing a second 90-second reasoning call.
We ended up decoupling the initial byte deadline from chunk cadence: 180 seconds for the first token, then dropping to a tight 15-second socket timeout once the stream is actually moving.
That one belongs on the list. In our stack it wasn't first: we don't set a timeout on the model clients, and the official OpenAI and Anthropic Python SDKs both default to 600 seconds, so our own 90-second key ran out long before them. One correction for anyone copying the numbers: plain httpx defaults to 5 seconds, not 60, on every read including the first.
Your split is the right shape. Silence before the first token and silence between tokens are two different budgets, and raising the effort spends the first one.__