Originally published on Loop & Retry — field notes on building LLM agents that survive production.
The failure that hurts is the one that doesn't throw. A traditional service fails loudly: an exception, a 500, a stack trace, a red line on a dashboard. An agent fails quietly. It runs to completion, returns a confident answer, exits zero — and the answer is wrong, or it spent forty steps and \$3 to conclude it couldn't do the thing, or it looped politely until it hit a cap nobody's watching. The $200 postmortem was a loud failure I happened to catch because costs spiked. The expensive ones are the quiet failures you never labeled, because you can't alert on a category you don't record. And for subjective tasks, your LLM judge might be hiding failures that look like success in its biased measurements.
What to measure when your agent works covered the happy path. This is the inverse: what to measure when it doesn't, and how to know that it didn't.
Exceptions are the tip of the iceberg
Here's the trap. Your agent has a try/except at the top of the loop. Exceptions get logged, counted, alerted. Your error rate looks like 0.5% and everyone's happy. Meanwhile:
- The agent hit its step cap and returned whatever it had — no exception, just a truncated answer.
- A tool returned
{"results": []}and the agent treated empty as "done." - The model produced a plausible-looking answer that's factually wrong — a perfect run by every mechanical measure.
- The agent looped between two states for thirty steps, then gave up — loop drift, which raises no error at all.
None of those increment your exception counter. All of them are failures. Your real failure rate isn't 0.5%; it's 0.5% that you can see plus an unknown, larger number you can't. Step one is to make every run end in a labeled outcome, not just "exception or not."
A run-outcome taxonomy
Every agent run should terminate with an explicit, recorded outcome. Not a boolean — a category. The minimum useful set:
| Outcome | What happened | How you detect it |
|---|---|---|
success |
Task done, verified | A post-hoc check passed (see below) |
hard_error |
Exception, crash, unrecoverable tool failure | The one you already catch |
budget_exhausted |
Hit a step / token / time cap mid-task | The cap fired before a terminal state |
gave_up |
Agent declared it couldn't finish | Model emitted a "cannot complete" terminal action |
looped |
Repeated states without progress | Progress detector tripped (loop drift) |
wrong |
Completed, but the output is bad | Only visible after the fact — sampling or user signal |
The point of the taxonomy is that these have different fixes. budget_exhausted means your caps are too tight or your task is too big — raise the cap or decompose. gave_up means a capability or tool gap — the agent knew it was stuck, which is the good failure. looped means your loop lacks a progress check. wrong is the dangerous one, because it's indistinguishable from success at runtime. Collapsing all of these into "error rate" throws away exactly the information that tells you what to do.
Continue reading: the rest of this post — including the sections that follow — is on Loop & Retry: read the full article
Top comments (0)