When moving tool-calling AI agents from a local demo into production, the failures that actually page you are rarely "the model wasn't smart enough."
Across recent production systems, three failure modes keep showing up:
- Timeout-after-commit on write tools: The HTTP request times out after the downstream API already created the ticket, sent the email, or wrote the row—and a naive retry duplicates the side effect.
- Silent retry loops: An agent hits an empty or partial result and calls the exact same tool with identical arguments until it burns the token budget.
- Context compaction drift: Summarizing old summaries drops negative constraints ("do not touch table X"), causing the agent to re-propose rejected paths 50 turns later.
Question for engineers running agents in production: Which failure mode hit your team first, and what deterministic guardrail (idempotency keys, state machines, external state ledgers, or hard circuit breakers) actually stopped it?
Top comments (3)
Timeout-after-commit is the one I'd test first, because it's the failure where the system looks like it failed and the side effect already happened.
One check you can run in staging: let a write tool succeed downstream, then drop the response before it reaches the agent, and watch what the agent does next. If it retries the write or tells the user the action failed, the timeout is being treated as a failure instead of an unknown outcome — and that's the exact gap to close before it pages you in production.
For us it was #1, timeout-after-commit. What fixed it was boring and deterministic. Every write tool takes an intent ID that the orchestrator generates before the call, and the downstream side stores it with the result. A retry with the same ID gets back the original result instead of a second ticket. Checking an expected version separately catches the other case, a stale read. For #2 a hard cap on identical (tool, args) pairs per run works better than any prompt instruction. For #3, keep negative constraints out of the summary entirely. Put them in a pinned "rules" block that compaction never touches.
The retry that lands twice. After that, idempotency keys stop being optional.