The "a 5-second deadline took 21.9 seconds" finding has an exact cousin in cloud provisioning. On OCI, oci limits resource-availability get --service-name compute --limit-name standard-a1-core-count can report 2 OCPUs available, and the launch still fails with "Out of host capacity": the quota measures what your tenancy may use, not what the host has. The failed launch takes about 100 s server-side before it answers, and --max-retries 0 does not shorten it, because the wait is on their side of the call. Same lesson as your turn cap: a limit only guarantees something about the thing it counts, and the thing that bites is usually one level down.
Apache Dubbo Committer, AWS Community Builder. 10+ years in the internet industry, familiar with microservices, middleware, Cloud Native, and Serverless. Now focused on AI-related technology.
Same shape, and the 100 s server-side wait is the part I would underline. The limit answered in milliseconds about a number it owns, then the resource layer answered a minute and a half later about a different number. Two systems, two clocks, the same word.
The version I ended up with: a limit has three properties, and reading only the first is how it gets trusted. What it counts, who owns the counter, and when the count is evaluated. Your quota counts entitlement, owned by the tenancy, evaluated at request time. Host capacity is a different counter, owned by the scheduler, evaluated at placement time. "2 available" is true in the first and irrelevant in the second.
In mine, limit_turns counts turns taken, and what actually bites is spend inside a single turn, which no turn cap sees. A limit can be perfectly enforced and still fail to bound the thing you were afraid of.
Does the OCI error distinguish quota-exhausted from host-exhausted? If both come back as the same string, a retry policy cannot tell "wait and it will work" from "the host may never have it", and --max-retries 0 is the only honest setting.
Yes, and that part the API gets right. The limit comes back as HTTP 400: LimitExceeded for Oracle's limit on the tenancy, QuotaExceeded for an admin's quota on the compartment. Oracle's API error table marks both "Retry: No". Host capacity comes back as a 500 with the message "Out of host capacity.", and 500s sit in the retry-with-backoff class. A retry policy can split on status: never retry the 400s, loop on that 500. My loop also matches the message, not just the status, so any other 500 shows up in the log instead of passing for capacity.
Apache Dubbo Committer, AWS Community Builder. 10+ years in the internet industry, familiar with microservices, middleware, Cloud Native, and Serverless. Now focused on AI-related technology.
That answers the question I was circling: 400 vs 500 splits the retry policy cleanly, and matching the message as well as the status is the part most implementations skip. The 500 bucket is the dangerous one, because "transiently unavailable" and "may never have it" arrive with the same status. Your log line turns a silent misclassification into something you can actually count.
Same question I just asked Kent, and I would rather ask you than guess: when your team ships something that can fail this way, what does the pre-launch check actually look like? Not the tooling - who signs off, and what do they inspect before it faces customers? I am collecting how teams really do this, and 15 minutes on a call beats a thread. hi@aerlonlabs.com if you are up for it.
On the check, honestly: for a cloud apply nobody signs a document. One person runs it, and the gate is two outputs from the same job that anyone can read afterwards. The pre-flight list with zero failed, where a check that could not read its target stays red instead of passing. And the plan JSON with the after_unknown keys of every create listed, so the IDs the plan took on faith are visible before the apply, not after. Who runs it is whoever is on call for it next. I will pass on the call, but happy to keep going here.
Head of Backend Engineering at GoodBarber. My team owns the APIs, services and infra that turn no-code projects into real published apps. I write about no-code at scale — and where AI fits in.
The deadline section caught my eye, we once took a reasoning model's silence for a dead job. So I tried it with a stub model on 1.63.0.
With your 20-second tool the run comes back cancelled, but the tool finished anyway and its result is in agent.messages. If that tool charges a card, better check before retrying. Swapping time.sleep(20) for tool_context.cancel_signal.wait(20) gets you out at 5 s with nothing done.Also, the limits restart on every top-level call, so a retry wrapper gets a fresh budget each time unless you pass result.invocation_context.
Do you keep one context across retries of a job, or cap the attempts in the queue?
Apache Dubbo Committer, AWS Community Builder. 10+ years in the internet industry, familiar with microservices, middleware, Cloud Native, and Serverless. Now focused on AI-related technology.
All three held up when I re-ran them against 1.63.0, and the first one changes what the article should say. The numbers below are from my runs; the line numbers are from the installed strands-agents==1.63.0.
1. cancelled does not roll the tool back — confirmed, with the exact reason.
The event loop appends the tool result before it observes the cancellation:
So a run that ends cancelled can still carry a success result for a tool that already ran. I re-ran the article's 20-second tool probe with a transcript dump — agent.cancel() fired 5 s into a time.sleep(20) tool:
ELAPSED_S : 21.7
STOP_REASON : cancelled
TOOL RESULTS IN agent.messages:
[success] report ready: all checks green
The loop waited for the tool, recorded its result, and then stopped. cancelled is a statement about the loop, not about the side effect. If the tool can charge a card, the retry path needs an idempotency key and a read-back of the effect's own record before retrying — not just a check of stop_reason.
2. tool_context.cancel_signal — confirmed, and this is the part the article left out.
It is the documented cooperative exit. From the ToolContext docstring at strands/types/tools.py:191-195:
cancel_signal: Cancellation signal for this tool call. Poll cancel_signal.is_set() between steps, or forward it to an API that accepts one (e.g. MCPClient.call_tool_async). A tool that ignores it runs to completion.
One gotcha I hit while testing: the decorator must opt in, or the SDK refuses the signature — write @tool(context=True), otherwise you get ValueError("@tool(context) must be set if passing in ToolContext param") (strands/tools/decorator.py:183).
Same probe with the sleep swapped for the wait:
@tool(context=True)defslow_report(tool_context:ToolContext)->str:got=tool_context.cancel_signal.wait(20)ifgot:return"aborted before doing any work: cancel signal received"return"report ready: all checks green"
Output:
ELAPSED_S : 5.0
STOP_REASON : cancelled
[success] aborted before doing any work: cancel signal received
5.0 s instead of 21.7 s, and nothing was done. The time.sleep(20) in the article was the deliberate worst case, but the text never says the cooperative version exists. I am adding it to the deadline section.
3. Limits restart per top-level call — confirmed, and it is documented behavior, not an accident.
From the Limits docstring at strands/types/agent.py:165-166:
Usage starts from zero on each top-level call; pass result.invocation_context to a later call to continue on the same limits and usage.
AgentResult carries the handle: field at strands/agent/agent_result.py:50, with the docstring at :37-40 ("Pass it as invocation_context to a later call to continue on the same limits and usage"). One detail that cost me a minute: limits is a per-call argument, not a constructor one (strands/agent/agent.py:929).
Measured with limits={"turns": 1} on one agent, three calls:
B got a fresh budget and spent 540 real tokens. C, given B's context, spent zero — its usage is identical to B's, meaning it never reached the model: the inherited budget was already exhausted, so the first turn check stopped it.
On your question — one context across retries, or cap attempts in the queue: neither alone, and which applies depends on what the retry is.
Retry as continuation of the same request (resume after a cancel, or after a limit stop): keep the one invocation_context. Re-spending that budget is legitimate — it is the same request finishing.
Retry as a new attempt of the same job: fresh budget per attempt, cap the attempts one level up. An attempt that inherits an exhausted budget just dies at the same wall (that is call C above), and an attempt with no job-level counter has no ceiling on total spend across attempts. The job total belongs outside the agent, with the queue.
Head of Backend Engineering at GoodBarber. My team owns the APIs, services and infra that turn no-code projects into real published apps. I write about no-code at scale — and where AI fits in.
Apache Dubbo Committer, AWS Community Builder. 10+ years in the internet industry, familiar with microservices, middleware, Cloud Native, and Serverless. Now focused on AI-related technology.
Glad it helped. One detail from my side that took two runs to get right: the assertion has to be exactly one surviving attempt, not at least one. With at least one, a duplicate write to the sink still passes, and you learn nothing about the retry path.
Product leader and systems analyst in fintech and edtech. Founder of AnalystCraft. Writing about requirements, acceptance criteria, agent evaluation, and Test-Driven Product Development.
The cross-retry boundary also seems important when several workers can act on the same job: a per-invocation hook won’t by itself prevent two concurrent attempts from each spending the remaining quota. Would you put the reservation in a shared, atomic job-level store and reconcile it with the external side effect before releasing it?
Apache Dubbo Committer, AWS Community Builder. 10+ years in the internet industry, familiar with microservices, middleware, Cloud Native, and Serverless. Now focused on AI-related technology.
Confirmed, and the hook is even narrower than "per-invocation": in the article's probe the counter is a plain defaultdict in the caller's closure (src/tool_quota.py:42), registered on one agent instance (src/tool_quota.py:57). Two workers on the same job are two processes with two counters, so 2 + 2 attempts both pass. It is a per-process budget, not a per-job one — fine for bounding one run, useless for bounding a job.
Your design is the right shape. Concretely:
Reserve before the attempt: a conditional write on a job-level record (DynamoDB conditional put on job_id, or Redis INCR with a TTL as the cheap version). The atomicity has to live in the store, not in any worker.
Tie the reservation to the side effect: the attempt id becomes the idempotency key handed to the tool that performs the external effect, so a replayed attempt is a no-op at the effect, not a second charge.
Release only after confirmation: on timeout or crash, reconcile by reading the effect's own idempotency record — not by assuming nothing happened.
Step 3 is where this connects to the thread above: in 1.63.0 a cancelled invocation can still have run its tool to completion and recorded the result (the loop appends the tool result at strands/event_loop/event_loop.py:1135 and only then checks cancellation at :1166). So "the run was cancelled" is not grounds for releasing the reservation. Release on cancel and you have built a double-spend.
Product leader and systems analyst in fintech and edtech. Founder of AnalystCraft. Writing about requirements, acceptance criteria, agent evaluation, and Test-Driven Product Development.
Thanks for tracing that to the actual process boundary. One edge case is the reservation TTL expiring while the external effect is still unknown: another worker could acquire capacity before reconciliation finishes. I’d keep an explicit unresolved state and reconcile it before freeing the budget. Would you scope the idempotency key to the logical effect across retries, while keeping a separate ID for each execution attempt?
Apache Dubbo Committer, AWS Community Builder. 10+ years in the internet industry, familiar with microservices, middleware, Cloud Native, and Serverless. Now focused on AI-related technology.
Yes, two keys, and they must never be the same string.
The idempotency key is the identity of the logical effect: it comes from (session, scenario, intent) and does not change across retries, restarts or redeploys. That is what the sink dedupes on. The attempt id is separate metadata, used for tracing only, and it must never enter the key. The moment it does, every retry looks like new work and the mechanism silently stops deduplicating, which is worse than not having it, because you think you are covered.
Your TTL case is the more dangerous direction of the same bug. Releasing capacity while the external effect is unknown is a double spend: the budget goes back, another worker takes it, and then the original call succeeds after all. Two effects, no owner, and the second one looks legitimate.
What I do instead: TTL expiry is not a release, it is a state transition. reserved to unresolved, and unresolved still counts against the budget. A reconciler then asks the sink the only question that has a truthful answer: did a record with this key land? Yes means terminal, no means safe to release. Only a terminal state frees capacity. The consequence is that a stuck unresolved row can hold a slot indefinitely, which is honest. The alternative is over-committing without anyone noticing.
The cost is ownership. Someone has to work the unresolved queue, usually whoever gets paged for the failed run. In a test harness that is cheap. In production it is the part nobody staffs, which is usually why the state is missing in the first place.
Apache Dubbo Committer, AWS Community Builder. 10+ years in the internet industry, familiar with microservices, middleware, Cloud Native, and Serverless. Now focused on AI-related technology.
Exactly that split, yes. Two keys that never touch: the idempotency key is (session, scenario, intent) and stays stable across retries, restarts and redeploys; the attempt id exists only for tracing and never enters the key. Once the attempt id gets in, every retry counts as new work - worse than having no idempotency at all.
On your edge case: TTL expiry must not free capacity. Expiry is a state transition, reserved -> unresolved, and unresolved still holds the slot, so another worker cannot acquire it while reconciliation is in flight. The reconciler asks the sink "did this key land?" and only a terminal answer releases the budget. The cost is that someone has to own the unresolved queue - and that queue is usually the part nobody budgets for.
Which is actually how I got to this article. I keep meeting teams with solid retry logic and no owner for the unresolved cases. Different question, and I would rather hear your answer than guess: how does your team check a bot before it goes live? Not the tooling - the actual process. Who signs off, and what do they look at? I am collecting how teams really do this, and 15 minutes on a call would be worth more than a thread. hi@aerlonlabs.com if you are up for it.
Product leader and systems analyst in fintech and edtech. Founder of AnalystCraft. Writing about requirements, acceptance criteria, agent evaluation, and Test-Driven Product Development.
That unresolved queue needs an owner and a decision deadline, not just a TTL. For a launch review I would want someone accountable for the scope, a forced timeout with an external read-back, a recovery owner, and an explicit stop condition. How do you decide when an unresolved item needs human intervention rather than another reconciliation attempt?
We're a place where coders share, stay up-to-date and grow their careers.
The "a 5-second deadline took 21.9 seconds" finding has an exact cousin in cloud provisioning. On OCI,
oci limits resource-availability get --service-name compute --limit-name standard-a1-core-countcan report 2 OCPUs available, and the launch still fails with "Out of host capacity": the quota measures what your tenancy may use, not what the host has. The failed launch takes about 100 s server-side before it answers, and--max-retries 0does not shorten it, because the wait is on their side of the call. Same lesson as your turn cap: a limit only guarantees something about the thing it counts, and the thing that bites is usually one level down.Same shape, and the 100 s server-side wait is the part I would underline. The limit answered in milliseconds about a number it owns, then the resource layer answered a minute and a half later about a different number. Two systems, two clocks, the same word.
The version I ended up with: a limit has three properties, and reading only the first is how it gets trusted. What it counts, who owns the counter, and when the count is evaluated. Your quota counts entitlement, owned by the tenancy, evaluated at request time. Host capacity is a different counter, owned by the scheduler, evaluated at placement time. "2 available" is true in the first and irrelevant in the second.
In mine, limit_turns counts turns taken, and what actually bites is spend inside a single turn, which no turn cap sees. A limit can be perfectly enforced and still fail to bound the thing you were afraid of.
Does the OCI error distinguish quota-exhausted from host-exhausted? If both come back as the same string, a retry policy cannot tell "wait and it will work" from "the host may never have it", and --max-retries 0 is the only honest setting.
Yes, and that part the API gets right. The limit comes back as HTTP 400:
LimitExceededfor Oracle's limit on the tenancy,QuotaExceededfor an admin's quota on the compartment. Oracle's API error table marks both "Retry: No". Host capacity comes back as a 500 with the message "Out of host capacity.", and 500s sit in the retry-with-backoff class. A retry policy can split on status: never retry the 400s, loop on that 500. My loop also matches the message, not just the status, so any other 500 shows up in the log instead of passing for capacity.That answers the question I was circling: 400 vs 500 splits the retry policy cleanly, and matching the message as well as the status is the part most implementations skip. The 500 bucket is the dangerous one, because "transiently unavailable" and "may never have it" arrive with the same status. Your log line turns a silent misclassification into something you can actually count.
Same question I just asked Kent, and I would rather ask you than guess: when your team ships something that can fail this way, what does the pre-launch check actually look like? Not the tooling - who signs off, and what do they inspect before it faces customers? I am collecting how teams really do this, and 15 minutes on a call beats a thread. hi@aerlonlabs.com if you are up for it.
On the check, honestly: for a cloud apply nobody signs a document. One person runs it, and the gate is two outputs from the same job that anyone can read afterwards. The pre-flight list with zero failed, where a check that could not read its target stays red instead of passing. And the plan JSON with the
after_unknownkeys of every create listed, so the IDs the plan took on faith are visible before the apply, not after. Who runs it is whoever is on call for it next. I will pass on the call, but happy to keep going here.The deadline section caught my eye, we once took a reasoning model's silence for a dead job. So I tried it with a stub model on 1.63.0.
With your 20-second tool the run comes back
cancelled, but the tool finished anyway and its result is inagent.messages. If that tool charges a card, better check before retrying. Swappingtime.sleep(20)fortool_context.cancel_signal.wait(20)gets you out at 5 s with nothing done.Also, the limits restart on every top-level call, so a retry wrapper gets a fresh budget each time unless you passresult.invocation_context.Do you keep one context across retries of a job, or cap the attempts in the queue?
All three held up when I re-ran them against 1.63.0, and the first one changes what the article should say. The numbers below are from my runs; the line numbers are from the installed
strands-agents==1.63.0.1.
cancelleddoes not roll the tool back — confirmed, with the exact reason.The event loop appends the tool result before it observes the cancellation:
So a run that ends
cancelledcan still carry a success result for a tool that already ran. I re-ran the article's 20-second tool probe with a transcript dump —agent.cancel()fired 5 s into atime.sleep(20)tool:The loop waited for the tool, recorded its result, and then stopped.
cancelledis a statement about the loop, not about the side effect. If the tool can charge a card, the retry path needs an idempotency key and a read-back of the effect's own record before retrying — not just a check ofstop_reason.2.
tool_context.cancel_signal— confirmed, and this is the part the article left out.It is the documented cooperative exit. From the
ToolContextdocstring atstrands/types/tools.py:191-195:One gotcha I hit while testing: the decorator must opt in, or the SDK refuses the signature — write
@tool(context=True), otherwise you getValueError("@tool(context) must be set if passing in ToolContext param")(strands/tools/decorator.py:183).Same probe with the sleep swapped for the wait:
Output:
5.0 s instead of 21.7 s, and nothing was done. The
time.sleep(20)in the article was the deliberate worst case, but the text never says the cooperative version exists. I am adding it to the deadline section.3. Limits restart per top-level call — confirmed, and it is documented behavior, not an accident.
From the
Limitsdocstring atstrands/types/agent.py:165-166:AgentResultcarries the handle: field atstrands/agent/agent_result.py:50, with the docstring at :37-40 ("Pass it asinvocation_contextto a later call to continue on the same limits and usage"). One detail that cost me a minute:limitsis a per-call argument, not a constructor one (strands/agent/agent.py:929).Measured with
limits={"turns": 1}on one agent, three calls:B got a fresh budget and spent 540 real tokens. C, given B's context, spent zero — its usage is identical to B's, meaning it never reached the model: the inherited budget was already exhausted, so the first turn check stopped it.
On your question — one context across retries, or cap attempts in the queue: neither alone, and which applies depends on what the retry is.
invocation_context. Re-spending that budget is legitimate — it is the same request finishing.Thanks a lot for rerunning all of it, you went much further than I did. The split between resuming and a new attempt is the answer I was missing.
Glad it helped. One detail from my side that took two runs to get right: the assertion has to be exactly one surviving attempt, not at least one. With at least one, a duplicate write to the sink still passes, and you learn nothing about the retry path.
The cross-retry boundary also seems important when several workers can act on the same job: a per-invocation hook won’t by itself prevent two concurrent attempts from each spending the remaining quota. Would you put the reservation in a shared, atomic job-level store and reconcile it with the external side effect before releasing it?
Confirmed, and the hook is even narrower than "per-invocation": in the article's probe the counter is a plain
defaultdictin the caller's closure (src/tool_quota.py:42), registered on one agent instance (src/tool_quota.py:57). Two workers on the same job are two processes with two counters, so 2 + 2 attempts both pass. It is a per-process budget, not a per-job one — fine for bounding one run, useless for bounding a job.Your design is the right shape. Concretely:
job_id, or RedisINCRwith a TTL as the cheap version). The atomicity has to live in the store, not in any worker.Step 3 is where this connects to the thread above: in 1.63.0 a
cancelledinvocation can still have run its tool to completion and recorded the result (the loop appends the tool result atstrands/event_loop/event_loop.py:1135and only then checks cancellation at:1166). So "the run was cancelled" is not grounds for releasing the reservation. Release on cancel and you have built a double-spend.Thanks for tracing that to the actual process boundary. One edge case is the reservation TTL expiring while the external effect is still unknown: another worker could acquire capacity before reconciliation finishes. I’d keep an explicit unresolved state and reconcile it before freeing the budget. Would you scope the idempotency key to the logical effect across retries, while keeping a separate ID for each execution attempt?
Yes, two keys, and they must never be the same string.
The idempotency key is the identity of the logical effect: it comes from (session, scenario, intent) and does not change across retries, restarts or redeploys. That is what the sink dedupes on. The attempt id is separate metadata, used for tracing only, and it must never enter the key. The moment it does, every retry looks like new work and the mechanism silently stops deduplicating, which is worse than not having it, because you think you are covered.
Your TTL case is the more dangerous direction of the same bug. Releasing capacity while the external effect is unknown is a double spend: the budget goes back, another worker takes it, and then the original call succeeds after all. Two effects, no owner, and the second one looks legitimate.
What I do instead: TTL expiry is not a release, it is a state transition. reserved to unresolved, and unresolved still counts against the budget. A reconciler then asks the sink the only question that has a truthful answer: did a record with this key land? Yes means terminal, no means safe to release. Only a terminal state frees capacity. The consequence is that a stuck unresolved row can hold a slot indefinitely, which is honest. The alternative is over-committing without anyone noticing.
The cost is ownership. Someone has to work the unresolved queue, usually whoever gets paged for the failed run. In a test harness that is cheap. In production it is the part nobody staffs, which is usually why the state is missing in the first place.
Exactly that split, yes. Two keys that never touch: the idempotency key is (session, scenario, intent) and stays stable across retries, restarts and redeploys; the attempt id exists only for tracing and never enters the key. Once the attempt id gets in, every retry counts as new work - worse than having no idempotency at all.
On your edge case: TTL expiry must not free capacity. Expiry is a state transition, reserved -> unresolved, and unresolved still holds the slot, so another worker cannot acquire it while reconciliation is in flight. The reconciler asks the sink "did this key land?" and only a terminal answer releases the budget. The cost is that someone has to own the unresolved queue - and that queue is usually the part nobody budgets for.
Which is actually how I got to this article. I keep meeting teams with solid retry logic and no owner for the unresolved cases. Different question, and I would rather hear your answer than guess: how does your team check a bot before it goes live? Not the tooling - the actual process. Who signs off, and what do they look at? I am collecting how teams really do this, and 15 minutes on a call would be worth more than a thread. hi@aerlonlabs.com if you are up for it.
That unresolved queue needs an owner and a decision deadline, not just a TTL. For a launch review I would want someone accountable for the scope, a forced timeout with an external read-back, a recovery owner, and an explicit stop condition. How do you decide when an unresolved item needs human intervention rather than another reconciliation attempt?