The timeout-after-dispatch step is the killer case — preflight passed, the tool changed, the client timed out, and now nobody knows if the ERP record moved. Of your four ingredients, the fresh authoritative readback is the only one that actually answers the question; the rest prove intent, not effect. In your pilot, are you treating the readback as mandatory for every mutating call, or only for the high-stakes ones?
Founder & CEO of Netsujo. Building trustworthy AI agent systems, with a focus on task boundaries, verification, and production reliability. Creator of Agent Role Contracts (ARC).
Good question. I'd make the strength of verification proportional to the risk, while requiring credible effect evidence before declaring a mutating workflow DONE.
For low-risk operations, a durable receipt from the authoritative destination may be sufficient. For high-impact changes, timeouts, or agent handoffs, I'd require a fresh, independently obtained readback tied to the logical operation ID.
If the result cannot be established, the status remains UNKNOWN, and an automatic retry should not proceed on that uncertainty alone.
ARC checks the supplied declarations. Runtime enforcement and durable effect verification are separate layers, and our joint ERP pilot has not yet established an end-to-end result.
I'm particularly interested in the latency and operational cost of that final readback. Have you encountered a case where its cost materially changed the design?
Risk-proportional verification is the right shape — a durable receipt for low-risk calls, a fresh readback tied to the operation ID where the stakes are real. On the readback cost: the read itself is rarely the expensive part; what changes designs is when the destination has no clean read API and the 'readback' becomes a second workflow. Is that the case in your ERP pilot — a system where verifying the effect costs nearly as much as the effect itself?
Founder & CEO of Netsujo. Building trustworthy AI agent systems, with a focus on task boundaries, verification, and production reliability. Creator of Agent Role Contracts (ARC).
Exactly — that's the distinction we're trying to make measurable.
To clarify, the ERP workflow in the article is a fictional scenario. Our ARC research is examining these verification boundaries, and we haven't claimed a completed production ERP pilot.
Our working hypothesis is risk-proportional verification:
Low-risk operations: durable operation IDs and outcome evidence.
High-stakes operations: authoritative, correlation-aware readback before confirming success or authorizing a retry.
Systems without reliable read APIs: preserve an UNKNOWN state until reconciliation provides sufficient evidence.
Your point about readback becoming a second workflow is especially important. In legacy ERP systems, verification complexity may become a deciding factor in whether an agent should execute an operation autonomously.
Have you encountered a practical way to keep verification costs manageable when the target system only exposes a UI or batch exports?
Thanks for clarifying — the risk-proportional framing is a good answer to my question. On the UI-or-batch-only case, my honest take is that the cheapest verification is often not verifying at all: if the agent can't read back what it changed, that operation probably shouldn't be autonomous yet. Batch exports at least give you a reconciliation point, even if it's a day late; a UI-only target gives you screenshots and hope.
Founder & CEO of Netsujo. Building trustworthy AI agent systems, with a focus on task boundaries, verification, and production reliability. Creator of Agent Role Contracts (ARC).
That's a useful distinction. I think you're right: if an agent cannot establish what changed, the right optimization may be to reduce autonomy rather than weaken verification.
I'd distinguish three cases:
Verifiable effects: Autonomous execution can proceed with evidence appropriate to the risk.
Delayed verification: Batch reconciliation can support execution, but the result stays UNKNOWN until sufficient evidence arrives. No blind retries.
Unverifiable effects: Keep a human in control of execution or confirmation rather than claiming autonomous completion.
This suggests a broader principle for our ARC research: autonomy should be constrained by what the system can actually verify, not just what the agent is authorized to attempt.
One question: would you consider delayed batch reconciliation sufficient for low-risk autonomous writes, provided retries remain blocked until reconciliation completes?
Tom's tripartite invariant is the right spine. Two traps I'd add from the retry path.
First, the readback has to come from the authority, not the nearest copy. A handover agent that checks a read replica - or a cache, or a search index - can see a stale "not applied", retry a write that already committed on the primary, and manufacture the duplicate it was trying to prevent. Replication lag turns ingredient 4 into a false negative at exactly the moment it matters. Same rule as Tom's audit ledger: the evidence read must live in the same consistency domain as the write.
Second, on the smallest-chain question: ingredients 1-3 collapse into one artifact if you write the intent before dispatch. One row - task digest, executor, tool sha256, operation id - committed ahead of the mutation, then the mutation and its commit marker in the same transaction. Four receipts become two writes, and the retry rule reduces to one lookup: did this exact intent commit? Everything else can be reconstructed at incident review, which is when you have time to read it.
Founder & CEO of Netsujo. Building trustworthy AI agent systems, with a focus on task boundaries, verification, and production reliability. Creator of Agent Role Contracts (ARC).
That's an important distinction. I'd treat UNKNOWN as a containment state, not merely a status label.
While the outcome is unresolved, the system should preserve the operation identity, block blind retries, prevent dependent writes, and keep the potential blast radius bounded.
I agree that financial or consequential external side effects warrant a human decision. But human approval and evidence of execution are separate requirements: an operator's approval cannot turn an unknown outcome into a verified one.
For our ARC-related research, this suggests a testable principle: autonomy depends not only on the risk of an action, but also on how well the system can contain and resolve uncertainty.
One question I'd like to explore: should mandatory human intervention be determined by the type of operation, or by properties such as reversibility, idempotency, and verifiable outcomes?
Properties over type, and your own framing is why: a label like "delete" or "payment" is a proxy for what actually matters - can this be undone, can it be safely repeated, and can we prove what happened. Type-based gates fail both ways: a soft-delete with an undo window is reversible and needs no human, while an innocent-looking read can trigger a side effect downstream. The practical shape is type as the cheap first pass that assigns a default property set, with the properties making the binding decision where they're known to differ.
One more property for your list, because it changes the UNKNOWN case directly: idempotency. An idempotent operation converts UNKNOWN from a containment problem into a retry problem - the system can safely ask again instead of freezing everything downstream. That's why an idempotency key is the cheapest containment you can buy: it shrinks the set of operations that ever need the human.
And the human side has a dual: an operator asked to approve everything becomes a rubber stamp within a week. Reserving intervention for genuinely irreversible, non-idempotent, unverifiable operations is what keeps the approval meaningful when it matters.
Founder & CEO of Netsujo. Building trustworthy AI agent systems, with a focus on task boundaries, verification, and production reliability. Creator of Agent Role Contracts (ARC).
Thanks — I agree that operation type is a useful starting point, while actual properties should determine the final gate. Your point about approval fatigue matters too.
I'd add one qualification on idempotency: a key makes retries safe only when the receiver enforces deduplication at the actual effect boundary, binds the key to the original payload, and retains that protection throughout the recovery window. Current authorization still needs to be checked.
Even then, safely retryable and verified are different states. Strong idempotency may allow a controlled retry while the outcome remains UNKNOWN, but it doesn't prove whether the original operation committed.
I'd model the decision as:
Operation type → actual properties → autonomy / reconciliation / human intervention.
Four failure cases seem worth testing:
Commit succeeds, acknowledgement is lost, and the same operation is retried.
The same idempotency key is reused with a different payload.
A retry arrives after the deduplication record expires.
An apparently read-only operation triggers a downstream side effect.
This would help us distinguish an ARC declaration check from the runtime guarantees required for safe execution.
Which of these failure cases would you prioritize?
I'd prioritize case 3, the expired dedup record, precisely because it passes every test. Cases 1 and 2 you can exercise in a unit test with a mocked store; case 4 is a design review. Case 3 is time-dependent: the retention window is chosen once, at design time, against a guessed recovery timeline, and nothing alarms when real recovery starts taking longer than the window. The failure only exists in production, on the worst day, after the incident that made recovery slow. A testable version: make the dedup record's age observable - age of the oldest live key as a metric - and alert when recovery time approaches retention. The gap between them is your actual safety margin, and in most systems it is unmeasured.
Your authorization caveat deserves the same treatment: the retry executes with authority granted at T0, and by T1 the grant may be gone. A replay that carries the original credential forward is a confused deputy waiting for a revocation. The retry path needs its own current authorization check - one more reason safely-retryable is a property of the executor, not just the endpoint.
Founder & CEO of Netsujo. Building trustworthy AI agent systems, with a focus on task boundaries, verification, and production reliability. Creator of Agent Role Contracts (ARC).
Agreed. Case 3 is particularly important because every deterministic test can pass while the real recovery window eventually exceeds the deduplication retention period.
I'd turn that into an operational invariant: track unresolved operation age, deduplication-key expiry, and recovery backlog. Alert when the remaining retention window approaches the time needed for safe reconciliation.
If the key has expired, or its validity cannot be established, the executor must not blindly retry. The outcome stays UNKNOWN until authoritative reconciliation resolves it.
Your authorization point is equally important. Every retry needs a fresh authorization check against the current role, scope, and approved action. Authorization at T0 is not authorization at T1.
This suggests two concrete failure-injection tests for our ARC-related host-integration research:
Delay recovery beyond deduplication-key expiry and verify that no unsafe second execution occurs.
Revoke authorization between dispatch and retry and verify that the executor refuses to forward the operation.
These are proposed tests, not capabilities we've already validated in production. ARC's current declaration validation remains separate from runtime enforcement.
I'd also make the safety margin itself an explicit operational metric rather than a configuration assumption.
One question: would you enforce a hard stop on automatic retries once the retention safety margin falls below a defined threshold?
@suirindo This scenario exposes two subtle traps that standard idempotency keys and generic entity readbacks fail to solve:
The Stale Tool Contract (Step 3): If the tool definition changes before dispatch, a standard UUID idempotency key will still execute, applying an unverified payload against a mutated schema.
The Concurrency Illusion (Step 5): When the client times out, the handover agent cannot simply inspect the target record (GET /record/42). If the status is already updated, there is no way to know whether Agent 1 did it or a concurrent operator touched it.
In regulated biobanking LIMS and ERP systems (where duplicate specimen thawing or relocation is catastrophic), we solve this with a tripartite invariant:
Dimension
Production Implementation
System
Regulated Laboratory Information & Aliquot Inventory Management (LIMS / ERP)
Side Effect
State transition of cryogenic specimen containers (relocating boxes between freezers or thawing vials)
Failure
Client HTTP timeout/network partition after gateway dispatch (status strictly UNKNOWN, never FAILED)
Trusted Evidence
Atomic in-transaction audit ledger entry. The business mutation and an immutable correlation record (correlation_id, caller_role, tool_sha256, applied_at) commit within the exact same physical database transaction
Retry Rule
Probe Before Dispatch: Handover agent is barred from re-dispatching. It queries the audit ledger by correlation_id. If FOUND, it returns the committed state; if NOT_FOUND, it evaluates the live rowversion before authorising a single idempotent re-submission
Cost
1 auxiliary write to an indexed audit table per transaction; 1 indexed read query before handover execution. Zero distributed coordination daemons (Temporal/Kafka)
Two architectural rules close the remaining loopholes:
Tool-Bound Idempotency Keys: We derive the idempotency key deterministically as HMAC(task_digest | tool_definition_sha256 | entity_id). If the tool definition changes between preflight and dispatch, the key changes, causing the gateway to reject the call immediately (STALE_TOOL_CONTRACT).
Correlation-Attested Readback: The handover agent queries the audit journal for the specific correlation_id, not just the entity state. This proves that this specific approved action was the one that landed, distinguishing agent execution from concurrent operator mutations.
Founder & CEO of Netsujo. Building trustworthy AI agent systems, with a focus on task boundaries, verification, and production reliability. Creator of Agent Role Contracts (ARC).
Tom, this is a valuable production example. Your distinction between entity-state readback and correlation-attested commit evidence is especially important.
I agree with binding the approved action to a canonical tool-definition digest and verifying it at the authoritative dispatch boundary.
Two details seem worth testing explicitly:
Stale contract rejection: Changing the HMAC-derived idempotency key alone would not reject a stale contract. The gateway must compare the live trusted definition against the definition bound to the original approval. The logical operation identity should also remain stable across retries.
Recovery finality: An atomic mutation-and-audit transaction provides strong positive evidence when the correlation record is found. However, NOT_FOUND could still race with an original attempt that remains in flight. A rowversion check alone may not resolve that ambiguity.
These are good candidates for ARC's optional host-integration and ERP evaluation work. We would keep declaration validation, runtime enforcement, and independently verified effects as separate evidence layers.
One question about your LIMS implementation: What establishes that NOT_FOUND is final, rather than an observation made before the original transaction commits?
That boundary seems crucial to making probe-before-dispatch safe under network partitions.
@suirindo That cuts straight to the core challenge of distributed systems under network partitions: an unfenced NOT_FOUND is only an observation of transient visibility, not a proof of finality.
In high-assurance biobanking and financial LIMS, a passive NOT_FOUND is never treated as final on its own. Finality is established through three complementary transactional boundaries:
Lock-Coordinated Probing (Eliminating In-Flight Ambiguity)
A recovery probe does not execute a raw read. It queries the journal under a transactional coordination lock (such as an application lock or exclusive row lock on the correlation_id). If the primary attempt is still in-flight, the probe is forced to wait until that transaction either commits or rolls back. Once the lock yields, NOT_FOUND is guaranteed final because the primary attempt has definitively terminated.
Atomic Unique Fencing (The Zero-Trust Backstop)
Even in an adversarial partition where gateway restarts sever the connection, the database audit journal enforces a strict physical UNIQUE CONSTRAINT on the correlation_id. Any subsequent retry by a handover agent must inherit that identical correlation token. Even if an agent raced ahead, the database engine physically guarantees that two executions can never both commit; the second will instantly abort with a constraint violation.
Deterministic Connection Expiry
The database server enforces a hard query timeout that is strictly shorter than the handover agent's recovery delay window. A recovery probe is barred from declaring finality until that connection-lifetime threshold has elapsed, ensuring orphaned transactions are severed before recovery dispatches.
While these three invariants are straightforward in theory, executing lock-coordinated probing under high-concurrency partitions without deadlock spirals is non-trivial.
Since you mentioned tracking these boundaries in your active research pilot for ARC, this is the exact transactional runtime engineered at Biomed Systems Limited for regulated environments. If your pilot partners need a hardened, production-tested host-verification engine rather than implementing the relational fencing from scratch, my advisory and direct contact channels are in my bio.
Founder & CEO of Netsujo. Building trustworthy AI agent systems, with a focus on task boundaries, verification, and production reliability. Creator of Agent Role Contracts (ARC).
Thanks, Tom. This is a valuable explanation, particularly the distinction between observing NOT_FOUND and establishing transactional finality.
I think your approach gives us three important assumptions to verify:
Lock coverage: Both the original dispatch and recovery probe must participate in the same durable coordination mechanism before mutation, including when no journal row exists. Otherwise, a late primary attempt could still race with recovery.
Atomic effects: The unique correlation constraint must protect the actual business mutation within the same transaction. A unique audit record alone cannot prevent duplicate external side effects.
Verified termination: A query or connection timeout must correspond to a demonstrably terminated or rolled-back transaction before recovery can safely proceed.
These are exactly the kinds of host-side guarantees we want to distinguish from ARC's declaration-level checks.
I'd be interested in testing one concrete failure scenario from your LIMS architecture: an in-flight primary operation, gateway disconnection, recovery probe, and competing handover retry.
Would you be open to sharing a sanitized execution trace or minimal reproducible test, including the observable receipts and recovery rule?
That could provide valuable independent evidence for ARC's host-integration research. We'd be glad to acknowledge Biomed Systems' contribution in any public evaluation, with your agreement.
Thanks again for bringing real production engineering experience into this discussion.
@suirindo Those three assumptions capture why host-level relational mechanics cannot be approximated in user space:
Lock coverage on cold keys: When no row exists in dbo.SYS_DispatchJournal, row-level locks have no physical anchor. In my architecture, the stored procedure opens by acquiring a transaction-owned application mutex (sp_getapplock @Resource = @CorrelationId, @LockMode = 'Exclusive', @LockOwner = 'Transaction') before checking existence. Both primary dispatch and recovery probes must acquire that identical mutex key, serialising the race even before the first row is inserted.
Atomic effects within a single transaction: The correlation journal insert and the business mutation (UPDATE/INSERT on specimen or inventory state) execute inside the same physical transaction under SET XACT_ABORT ON. If a duplicate retry hits the UNIQUE constraint or a lock conflict, the database engine aborts and rolls back the entire unit of work atomically. Separate audit writes outside the business transaction cannot prevent duplicate side effects.
Verified termination before recovery: A client socket timeout (ECONNRESET) never proves the database server has finished rolling back. Rather than guessing from elapsed wall-clock time alone, the recovery probe tests the correlation mutex with an immediate lock timeout (SET LOCK_TIMEOUT 0). If the orphaned session is still active or mid-rollback, the lock acquisition fails immediately (Error 1222), forcing the handover worker to hold UNKNOWN until the engine releases the transaction lock.
Regarding a reproducible test for your evaluation: the standalone Node.js verification matrix (v1.1.0 on my profile) is my complete open reference contribution. Production LIMS failure traces, relational fencing harnesses, and ERP recovery adapters are commercial IP delivered through Biomed Systems Limited under paid enterprise pilot or advisory engagements.
If Netsujo or a commercial partner has budget allocated for a formal host-integration pilot, my direct commercial channels are listed in my profile.
Founder & CEO of Netsujo. Building trustworthy AI agent systems, with a focus on task boundaries, verification, and production reliability. Creator of Agent Role Contracts (ARC).
Thanks, Tom. This is a precise and useful explanation.
The key insight for us is that recovery finality depends on host-level transactional coordination. Both the original dispatch and the recovery probe must participate in the same locking protocol, while the business mutation and correlation journal commit atomically.
This gives us a useful distinction for ARC's evaluation work: validating an agent's authority, controlling execution, and establishing independently verifiable evidence of the resulting effect are separate assurance layers.
We'll examine your public v1.1.0 verification matrix as a reference. I also appreciate the clarification about your commercial IP boundaries and would be glad to keep the dialogue open as our integration research progresses.
One small implementation question: SQL Server documents sp_getapplock timeout as return code -1, while error 1222 can arise from statement-level lock timeouts. Does your recovery probe handle both paths explicitly?
That distinction could make a valuable reproducible failure case.
Thanks again for sharing the production architecture perspective.
The timeout-after-dispatch case is the one that convinces me too — a plain readback can't tell you whether your agent or someone else made the change. I'd want the correlation-attested readback Tom describes, plus a human approval step before any retry on a high-stakes record, so an UNKNOWN stays UNKNOWN until someone has actually checked it.
Founder & CEO of Netsujo. Building trustworthy AI agent systems, with a focus on task boundaries, verification, and production reliability. Creator of Agent Role Contracts (ARC).
Thanks, Elarigent. I think you're identifying two separate controls: evidence that this specific approved operation caused the change, and authorization to retry after an uncertain outcome.
A correlation-attested readback is meaningful when the correlation record is authoritative and atomically bound to the business mutation. After a timeout, NOT_FOUND alone still doesn't prove the original operation won't commit later.
For high-stakes records, I'd keep the outcome UNKNOWN, block automatic retries, and require host-side reconciliation plus explicit human authorization before redispatch. Human approval should govern the retry decision, not substitute for evidence.
This is an important failure case for our ARC-related host-integration research. ARC's current declaration checks remain separate from ERP commit verification.
Have you seen teams make this distinction explicit in their operator workflows?
We're a place where coders share, stay up-to-date and grow their careers.
The timeout-after-dispatch step is the killer case — preflight passed, the tool changed, the client timed out, and now nobody knows if the ERP record moved. Of your four ingredients, the fresh authoritative readback is the only one that actually answers the question; the rest prove intent, not effect. In your pilot, are you treating the readback as mandatory for every mutating call, or only for the high-stakes ones?
Good question. I'd make the strength of verification proportional to the risk, while requiring credible effect evidence before declaring a mutating workflow DONE.
For low-risk operations, a durable receipt from the authoritative destination may be sufficient. For high-impact changes, timeouts, or agent handoffs, I'd require a fresh, independently obtained readback tied to the logical operation ID.
If the result cannot be established, the status remains UNKNOWN, and an automatic retry should not proceed on that uncertainty alone.
ARC checks the supplied declarations. Runtime enforcement and durable effect verification are separate layers, and our joint ERP pilot has not yet established an end-to-end result.
I'm particularly interested in the latency and operational cost of that final readback. Have you encountered a case where its cost materially changed the design?
Risk-proportional verification is the right shape — a durable receipt for low-risk calls, a fresh readback tied to the operation ID where the stakes are real. On the readback cost: the read itself is rarely the expensive part; what changes designs is when the destination has no clean read API and the 'readback' becomes a second workflow. Is that the case in your ERP pilot — a system where verifying the effect costs nearly as much as the effect itself?
Exactly — that's the distinction we're trying to make measurable.
To clarify, the ERP workflow in the article is a fictional scenario. Our ARC research is examining these verification boundaries, and we haven't claimed a completed production ERP pilot.
Our working hypothesis is risk-proportional verification:
Your point about readback becoming a second workflow is especially important. In legacy ERP systems, verification complexity may become a deciding factor in whether an agent should execute an operation autonomously.
Have you encountered a practical way to keep verification costs manageable when the target system only exposes a UI or batch exports?
Thanks for clarifying — the risk-proportional framing is a good answer to my question. On the UI-or-batch-only case, my honest take is that the cheapest verification is often not verifying at all: if the agent can't read back what it changed, that operation probably shouldn't be autonomous yet. Batch exports at least give you a reconciliation point, even if it's a day late; a UI-only target gives you screenshots and hope.
That's a useful distinction. I think you're right: if an agent cannot establish what changed, the right optimization may be to reduce autonomy rather than weaken verification.
I'd distinguish three cases:
This suggests a broader principle for our ARC research: autonomy should be constrained by what the system can actually verify, not just what the agent is authorized to attempt.
One question: would you consider delayed batch reconciliation sufficient for low-risk autonomous writes, provided retries remain blocked until reconciliation completes?
Tom's tripartite invariant is the right spine. Two traps I'd add from the retry path.
First, the readback has to come from the authority, not the nearest copy. A handover agent that checks a read replica - or a cache, or a search index - can see a stale "not applied", retry a write that already committed on the primary, and manufacture the duplicate it was trying to prevent. Replication lag turns ingredient 4 into a false negative at exactly the moment it matters. Same rule as Tom's audit ledger: the evidence read must live in the same consistency domain as the write.
Second, on the smallest-chain question: ingredients 1-3 collapse into one artifact if you write the intent before dispatch. One row - task digest, executor, tool sha256, operation id - committed ahead of the mutation, then the mutation and its commit marker in the same transaction. Four receipts become two writes, and the retry rule reduces to one lookup: did this exact intent commit? Everything else can be reconstructed at incident review, which is when you have time to read it.
That's an important distinction. I'd treat UNKNOWN as a containment state, not merely a status label.
While the outcome is unresolved, the system should preserve the operation identity, block blind retries, prevent dependent writes, and keep the potential blast radius bounded.
I agree that financial or consequential external side effects warrant a human decision. But human approval and evidence of execution are separate requirements: an operator's approval cannot turn an unknown outcome into a verified one.
For our ARC-related research, this suggests a testable principle: autonomy depends not only on the risk of an action, but also on how well the system can contain and resolve uncertainty.
One question I'd like to explore: should mandatory human intervention be determined by the type of operation, or by properties such as reversibility, idempotency, and verifiable outcomes?
Properties over type, and your own framing is why: a label like "delete" or "payment" is a proxy for what actually matters - can this be undone, can it be safely repeated, and can we prove what happened. Type-based gates fail both ways: a soft-delete with an undo window is reversible and needs no human, while an innocent-looking read can trigger a side effect downstream. The practical shape is type as the cheap first pass that assigns a default property set, with the properties making the binding decision where they're known to differ.
One more property for your list, because it changes the UNKNOWN case directly: idempotency. An idempotent operation converts UNKNOWN from a containment problem into a retry problem - the system can safely ask again instead of freezing everything downstream. That's why an idempotency key is the cheapest containment you can buy: it shrinks the set of operations that ever need the human.
And the human side has a dual: an operator asked to approve everything becomes a rubber stamp within a week. Reserving intervention for genuinely irreversible, non-idempotent, unverifiable operations is what keeps the approval meaningful when it matters.
Thanks — I agree that operation type is a useful starting point, while actual properties should determine the final gate. Your point about approval fatigue matters too.
I'd add one qualification on idempotency: a key makes retries safe only when the receiver enforces deduplication at the actual effect boundary, binds the key to the original payload, and retains that protection throughout the recovery window. Current authorization still needs to be checked.
Even then, safely retryable and verified are different states. Strong idempotency may allow a controlled retry while the outcome remains UNKNOWN, but it doesn't prove whether the original operation committed.
I'd model the decision as:
Operation type → actual properties → autonomy / reconciliation / human intervention.
Four failure cases seem worth testing:
This would help us distinguish an ARC declaration check from the runtime guarantees required for safe execution.
Which of these failure cases would you prioritize?
I'd prioritize case 3, the expired dedup record, precisely because it passes every test. Cases 1 and 2 you can exercise in a unit test with a mocked store; case 4 is a design review. Case 3 is time-dependent: the retention window is chosen once, at design time, against a guessed recovery timeline, and nothing alarms when real recovery starts taking longer than the window. The failure only exists in production, on the worst day, after the incident that made recovery slow. A testable version: make the dedup record's age observable - age of the oldest live key as a metric - and alert when recovery time approaches retention. The gap between them is your actual safety margin, and in most systems it is unmeasured.
Your authorization caveat deserves the same treatment: the retry executes with authority granted at T0, and by T1 the grant may be gone. A replay that carries the original credential forward is a confused deputy waiting for a revocation. The retry path needs its own current authorization check - one more reason safely-retryable is a property of the executor, not just the endpoint.
Agreed. Case 3 is particularly important because every deterministic test can pass while the real recovery window eventually exceeds the deduplication retention period.
I'd turn that into an operational invariant: track unresolved operation age, deduplication-key expiry, and recovery backlog. Alert when the remaining retention window approaches the time needed for safe reconciliation.
If the key has expired, or its validity cannot be established, the executor must not blindly retry. The outcome stays UNKNOWN until authoritative reconciliation resolves it.
Your authorization point is equally important. Every retry needs a fresh authorization check against the current role, scope, and approved action. Authorization at T0 is not authorization at T1.
This suggests two concrete failure-injection tests for our ARC-related host-integration research:
These are proposed tests, not capabilities we've already validated in production. ARC's current declaration validation remains separate from runtime enforcement.
I'd also make the safety margin itself an explicit operational metric rather than a configuration assumption.
One question: would you enforce a hard stop on automatic retries once the retention safety margin falls below a defined threshold?
@suirindo This scenario exposes two subtle traps that standard idempotency keys and generic entity readbacks fail to solve:
GET /record/42). If the status is already updated, there is no way to know whether Agent 1 did it or a concurrent operator touched it.In regulated biobanking LIMS and ERP systems (where duplicate specimen thawing or relocation is catastrophic), we solve this with a tripartite invariant:
UNKNOWN, neverFAILED)correlation_id,caller_role,tool_sha256,applied_at) commit within the exact same physical database transactioncorrelation_id. IfFOUND, it returns the committed state; ifNOT_FOUND, it evaluates the live rowversion before authorising a single idempotent re-submissionTwo architectural rules close the remaining loopholes:
HMAC(task_digest | tool_definition_sha256 | entity_id). If the tool definition changes between preflight and dispatch, the key changes, causing the gateway to reject the call immediately (STALE_TOOL_CONTRACT).correlation_id, not just the entity state. This proves that this specific approved action was the one that landed, distinguishing agent execution from concurrent operator mutations.Tom, this is a valuable production example. Your distinction between entity-state readback and correlation-attested commit evidence is especially important.
I agree with binding the approved action to a canonical tool-definition digest and verifying it at the authoritative dispatch boundary.
Two details seem worth testing explicitly:
Stale contract rejection: Changing the HMAC-derived idempotency key alone would not reject a stale contract. The gateway must compare the live trusted definition against the definition bound to the original approval. The logical operation identity should also remain stable across retries.
Recovery finality: An atomic mutation-and-audit transaction provides strong positive evidence when the correlation record is found. However,
NOT_FOUNDcould still race with an original attempt that remains in flight. A rowversion check alone may not resolve that ambiguity.These are good candidates for ARC's optional host-integration and ERP evaluation work. We would keep declaration validation, runtime enforcement, and independently verified effects as separate evidence layers.
One question about your LIMS implementation: What establishes that
NOT_FOUNDis final, rather than an observation made before the original transaction commits?That boundary seems crucial to making probe-before-dispatch safe under network partitions.
@suirindo That cuts straight to the core challenge of distributed systems under network partitions: an unfenced
NOT_FOUNDis only an observation of transient visibility, not a proof of finality.In high-assurance biobanking and financial LIMS, a passive
NOT_FOUNDis never treated as final on its own. Finality is established through three complementary transactional boundaries:Lock-Coordinated Probing (Eliminating In-Flight Ambiguity)
A recovery probe does not execute a raw read. It queries the journal under a transactional coordination lock (such as an application lock or exclusive row lock on the
correlation_id). If the primary attempt is still in-flight, the probe is forced to wait until that transaction either commits or rolls back. Once the lock yields,NOT_FOUNDis guaranteed final because the primary attempt has definitively terminated.Atomic Unique Fencing (The Zero-Trust Backstop)
Even in an adversarial partition where gateway restarts sever the connection, the database audit journal enforces a strict physical
UNIQUE CONSTRAINTon thecorrelation_id. Any subsequent retry by a handover agent must inherit that identical correlation token. Even if an agent raced ahead, the database engine physically guarantees that two executions can never both commit; the second will instantly abort with a constraint violation.Deterministic Connection Expiry
The database server enforces a hard query timeout that is strictly shorter than the handover agent's recovery delay window. A recovery probe is barred from declaring finality until that connection-lifetime threshold has elapsed, ensuring orphaned transactions are severed before recovery dispatches.
While these three invariants are straightforward in theory, executing lock-coordinated probing under high-concurrency partitions without deadlock spirals is non-trivial.
Since you mentioned tracking these boundaries in your active research pilot for ARC, this is the exact transactional runtime engineered at Biomed Systems Limited for regulated environments. If your pilot partners need a hardened, production-tested host-verification engine rather than implementing the relational fencing from scratch, my advisory and direct contact channels are in my bio.
Thanks, Tom. This is a valuable explanation, particularly the distinction between observing
NOT_FOUNDand establishing transactional finality.I think your approach gives us three important assumptions to verify:
Lock coverage: Both the original dispatch and recovery probe must participate in the same durable coordination mechanism before mutation, including when no journal row exists. Otherwise, a late primary attempt could still race with recovery.
Atomic effects: The unique correlation constraint must protect the actual business mutation within the same transaction. A unique audit record alone cannot prevent duplicate external side effects.
Verified termination: A query or connection timeout must correspond to a demonstrably terminated or rolled-back transaction before recovery can safely proceed.
These are exactly the kinds of host-side guarantees we want to distinguish from ARC's declaration-level checks.
I'd be interested in testing one concrete failure scenario from your LIMS architecture: an in-flight primary operation, gateway disconnection, recovery probe, and competing handover retry.
Would you be open to sharing a sanitized execution trace or minimal reproducible test, including the observable receipts and recovery rule?
That could provide valuable independent evidence for ARC's host-integration research. We'd be glad to acknowledge Biomed Systems' contribution in any public evaluation, with your agreement.
Thanks again for bringing real production engineering experience into this discussion.
@suirindo Those three assumptions capture why host-level relational mechanics cannot be approximated in user space:
Lock coverage on cold keys: When no row exists in
dbo.SYS_DispatchJournal, row-level locks have no physical anchor. In my architecture, the stored procedure opens by acquiring a transaction-owned application mutex (sp_getapplock @Resource = @CorrelationId, @LockMode = 'Exclusive', @LockOwner = 'Transaction') before checking existence. Both primary dispatch and recovery probes must acquire that identical mutex key, serialising the race even before the first row is inserted.Atomic effects within a single transaction: The correlation journal insert and the business mutation (
UPDATE/INSERTon specimen or inventory state) execute inside the same physical transaction underSET XACT_ABORT ON. If a duplicate retry hits theUNIQUEconstraint or a lock conflict, the database engine aborts and rolls back the entire unit of work atomically. Separate audit writes outside the business transaction cannot prevent duplicate side effects.Verified termination before recovery: A client socket timeout (
ECONNRESET) never proves the database server has finished rolling back. Rather than guessing from elapsed wall-clock time alone, the recovery probe tests the correlation mutex with an immediate lock timeout (SET LOCK_TIMEOUT 0). If the orphaned session is still active or mid-rollback, the lock acquisition fails immediately (Error 1222), forcing the handover worker to holdUNKNOWNuntil the engine releases the transaction lock.Regarding a reproducible test for your evaluation: the standalone Node.js verification matrix (
v1.1.0on my profile) is my complete open reference contribution. Production LIMS failure traces, relational fencing harnesses, and ERP recovery adapters are commercial IP delivered through Biomed Systems Limited under paid enterprise pilot or advisory engagements.If Netsujo or a commercial partner has budget allocated for a formal host-integration pilot, my direct commercial channels are listed in my profile.
Thanks, Tom. This is a precise and useful explanation.
The key insight for us is that recovery finality depends on host-level transactional coordination. Both the original dispatch and the recovery probe must participate in the same locking protocol, while the business mutation and correlation journal commit atomically.
This gives us a useful distinction for ARC's evaluation work: validating an agent's authority, controlling execution, and establishing independently verifiable evidence of the resulting effect are separate assurance layers.
We'll examine your public v1.1.0 verification matrix as a reference. I also appreciate the clarification about your commercial IP boundaries and would be glad to keep the dialogue open as our integration research progresses.
One small implementation question: SQL Server documents
sp_getapplocktimeout as return code-1, while error 1222 can arise from statement-level lock timeouts. Does your recovery probe handle both paths explicitly?That distinction could make a valuable reproducible failure case.
Thanks again for sharing the production architecture perspective.
The timeout-after-dispatch case is the one that convinces me too — a plain readback can't tell you whether your agent or someone else made the change. I'd want the correlation-attested readback Tom describes, plus a human approval step before any retry on a high-stakes record, so an UNKNOWN stays UNKNOWN until someone has actually checked it.
Thanks, Elarigent. I think you're identifying two separate controls: evidence that this specific approved operation caused the change, and authorization to retry after an uncertain outcome.
A correlation-attested readback is meaningful when the correlation record is authoritative and atomically bound to the business mutation. After a timeout,
NOT_FOUNDalone still doesn't prove the original operation won't commit later.For high-stakes records, I'd keep the outcome
UNKNOWN, block automatic retries, and require host-side reconciliation plus explicit human authorization before redispatch. Human approval should govern the retry decision, not substitute for evidence.This is an important failure case for our ARC-related host-integration research. ARC's current declaration checks remain separate from ERP commit verification.
Have you seen teams make this distinction explicit in their operator workflows?