The timeout that might have worked: how to retry when you don't know if the first call landed

Originally published on Loop & Retry — field notes on building LLM agents that survive production.

Most retry code is written for two outcomes: the call worked, or it failed. A real call has a third: you sent it, the connection died, and you have no idea whether the other side acted. The agent logs TimeoutError, the retry wrapper sees an exception, and it sends the request again. If the first request had in fact landed, you've now done the thing twice.

This is a practical post, not a definitional one. If you want the vocabulary, idempotency keys for agents covers how to derive a key from intent, and retry budgets covers how much retrying you can afford. This one is about the narrower question that comes up in a running system: the call timed out, the effect might exist, what do I do next?

Four outcomes, not two

Write the outcome type down explicitly, because the bug lives in the fact that most code doesn't have it.

from enum import Enum

class Outcome(Enum):
    OK = "ok"                  # got a success response
    REJECTED = "rejected"      # got a definite no (4xx, validation, auth)
    FAILED_BEFORE = "failed"   # provably never reached the effect (DNS, refused, TLS)
    UNKNOWN = "unknown"        # sent, then lost: timeout, reset, 5xx from a proxy

The distinction that matters is FAILED_BEFORE versus UNKNOWN. A connection-refused error means the server never saw you, so retrying is safe for any tool. A read timeout means the server may have seen you, done the work, and failed to answer. A 502 or 504 from a gateway is the same: the gateway gave up, the upstream may not have. Both look like "an error" in a stack trace and need opposite handling.

If your HTTP client collapses these into one exception class, the first job is to split them. In Python's requests and httpx, connect errors and read timeouts are different exception types. Use them.

Sort your tools into three classes

Before deciding what an UNKNOWN outcome should do, you need to know what the tool does. For every tool an agent can call, answer one question: what happens if this runs twice?

The third row is where the damage is. "Append to a log" feels harmless until the log feeds a billing job. "Post a comment" feels harmless until it's a customer-facing thread.

Two rules fall out of this:

1. Classification belongs to the tool, not the retry wrapper. The wrapper doesn't know that update_ticket is convergent and add_ticket_note accumulates. Put a retry_class field on the tool definition, and make the wrapper refuse to retry anything unclassified. Unclassified defaulting to "safe" is the mistake that ships.
2. Many accumulating tools can be made convergent, and that's usually the better fix. If the API takes an idempotency key, send one derived from the intent (ticket-481/refund/2026-11), and the tool becomes convergent. The wrapper then only has to be careful with tools that genuinely can't.

Verify before you write again

For an accumulating tool that has no idempotency key, an UNKNOWN outcome has exactly one correct next step: go look. Don't retry. Read the state the call was supposed to change, and decide from that.

def call_accumulating(tool, args, *, probe, attempts=3):
    """probe(args) -> True if the effect already exists, False if not,
    None if we can't tell."""
    for attempt in range(attempts):
        outcome, result = tool.invoke(args)

        if outcome is Outcome.OK:
            return result
        if outcome is Outcome.REJECTED:
            raise ToolRejected(result)      # a 400 stays a 400, don't retry it

        if outcome is Outcome.UNKNOWN:
            seen = probe(args)
            if seen is True:
                return tool.recover_result(args)   # it landed; don't resend
            if seen is None:
                raise NeedsHuman("effect unknown, probe inconclusive", args)
            # seen is False: it did not land, safe to loop and try again

        sleep_with_jitter(attempt)
    raise RetriesExhausted(args)

Three details are doing real work here.

The probe has to be able to say "I can't tell." A probe that returns False when the read itself fails will cause exactly the duplicate you were trying to avoid. Three-valued, not two.

The probe has to match on something the call controls. Searching "did an email go to this address recently?" matches the previous, legitimate email. Put a token you generate in the payload (a reference id in the subject line, a metadata field on the charge, a marker in the comment body) and probe for that token. This is the same move as a derived idempotency key, applied after the fact for systems that never offered one.

Mind replication lag. If the probe reads from a replica or a search index, a "not found" right after the timeout may just mean it hasn't propagated. For anything where a duplicate is expensive, wait out the known lag before probing, or probe the primary. Otherwise the probe becomes a source of false False.

When there's nothing to read

Some tools are write-only. You can send an SMS but you can't ask the carrier whether you did. For those, you can't verify, so the choice is between the two bad outcomes, and you should make it on purpose instead of by default.

• At-most-once: on UNKNOWN, don't retry, mark the step uncertain, and surface it. You might drop a message. You won't duplicate one.
• At-least-once: on UNKNOWN, retry. You might duplicate. You won't drop.

Choose per tool by asking which mistake is cheaper to explain. A duplicate "your order shipped" notification is an annoyance. A duplicate wire transfer is an incident. A dropped heartbeat ping doesn't matter at all. Write the choice into the tool's definition next to retry_class, so it's reviewed once instead of rediscovered during an outage.

Whichever you pick, keep a reconciliation record: an append-only line written before the call (intent_id, tool, args hash, status=sent) and updated after. When a step ends UNKNOWN, that record is what lets a later job, or a person, check the downstream system and close it out. An agent that retried silently leaves nothing to reconcile against.

What the agent should be told

A common failure is that the wrapper handles this correctly and the model undoes it. The tool returns "timeout", the model reads that as "it didn't work", and it calls the tool again on its own initiative. Your carefully gated wrapper is bypassed one layer up.

So change what the model sees. Don't return the raw exception. Return a structured result that states the uncertainty and the instruction:

{
  "status": "unknown",
  "message": "The request timed out. The email MAY have been sent.",
  "next": "Do not call send_email again. Call find_sent_email with ref=INV-481 to check.",
  "ref": "INV-481"
}

Telling the model the specific safe next action works better than telling it to be careful. It also gives the model something concrete to do, which stops it from improvising.

Testing it

You can test all of this without waiting for a real outage, by building a fake tool that misbehaves in the one way that matters: it performs the effect and then drops the response.

class LossyEmail:
    def __init__(self, drop_every=2):
        self.sent = []
        self.n = 0
        self.drop_every = drop_every

    def invoke(self, args):
        self.n += 1
        self.sent.append(args)                 # the effect always happens
        if self.n % self.drop_every == 0:
            return Outcome.UNKNOWN, None       # ...but we lose the answer
        return Outcome.OK, {"id": self.n}

Run your wrapper against it and assert the property you care about: the number of effects equals the number of intents, regardless of how many responses were dropped. A test that only checks "retries happened on timeout" will pass on the buggy version. A test that checks len(fake.sent) == len(intents) will not.

What I'd actually do

If you do only three things:

1. Split timeouts from connection failures in your error types, and give timeouts their own outcome. Treating them as ordinary failures is the root of this whole class of bug.
2. Put retry_class on every tool and have the retry wrapper refuse to retry an accumulating tool on UNKNOWN. Default unclassified tools to the cautious side.
3. Return "unknown, here's how to check" to the model instead of a raw error, so the retry decision isn't quietly re-made one level up.

Everything else here (probes, reconciliation records, per-tool at-most-once choices) is a refinement for the tools that can hurt you. Start with the three, and the next time something times out you'll be able to say whether it happened. That's more than most retry loops can do today.