An executable synthetic test for a subtle failure in AI-agent review evidence.
Two AI agents can approve the same pull request. They can even attach two different names to their reports. That still leaves a question: who actually had the authority to produce the review evidence?
In discussions following my posts on what an AI agent means by “done” and whether two AI reviewers are truly independent, TokenLat made a useful distinction: establish that the required evidence exists and is bound to the right candidate first; then test whether an unauthorized producer could make that evidence look independent.
I turned that question into a small, fully synthetic Node.js fixture. It uses no AI model, real repository, or production credentials.
The one-field problem
Imagine an evaluation event:
{
"id": "e-1",
"type": "evaluation",
"candidate": "sha256:synthetic-pr-head-a",
"claimedProducer": "independent-reviewer"
}
A naive checker might look for an evaluation event, compare the candidate, see independent-reviewer, and return PASS.
But what if the implementer emitted that exact JSON?
Changing a producer label is not a change of authority. If the same component can write both the alleged independent review and the receipt naming its author, the evidence is self-attestation.
Four cases, not just one happy path
I used one fixed candidate identity and three event types: declaration, evaluation, and acceptance. The fixture tests:
| Case | What changes? | Required verdict |
|---|---|---|
| 1. Complete | All three events exist and have matching, authorized provenance | PASS |
| 2. Missing evaluation | Remove the evaluation event | PENDING |
| 3. Wrong candidate | Bind the evaluation to a different candidate | DENY |
| 4. Spoofed reviewer | The executor emits an evaluation while claiming to be the independent reviewer | DENY |
The fourth test only makes sense after the first three: otherwise a rejection could simply be caused by a missing or mismatched record, not a failure to verify producer independence.
A claimed name is not an authenticated identity
The essential check is not:
receipt.claimedProducer === 'independent-reviewer'
It is closer to:
hostRecord.authenticatedRole === 'independent-reviewer' &&
hostRecord.candidate === receipt.candidate &&
hostRecord.eventDigest === digest(receipt)
The crucial distinction is where hostRecord comes from. It must come through a separately controlled, authenticated evidence channel. A producer-supplied field named authenticatedRole would accomplish nothing.
In my toy fixture, a mock host journal supplies that trusted provenance. The check is deliberately narrow: it demonstrates how a string-only verifier can wrongly accept a forged reviewer label, while a verifier using independent provenance rejects it. The mock does not provide real authentication, signature verification, filesystem isolation, or runtime enforcement.
I ran the four cases and one additional regression test against the naive checker. All five Node.js tests passed. In the spoof case, the string-only verifier returned PASS and the provenance-aware toy verifier returned DENY.
What this does—and does not—say about ARC
I maintain Agent Role Contracts (ARC), which checks the consistency of declared agent roles, task scopes, and review relationships. That is a declaration-level check.
An ARC declaration by itself does not authenticate the reviewer who produced an event, prevent an executor from writing to a review channel, or prove that the reviewed revision was the one merged. Those responsibilities belong to the host, authorization system, CI, and final acceptance process.
The same discipline applies to exemptions. “Independent review not required for this task” should be an explicit, authorized scope decision. Silence or a missing event is not an exemption.
The question I would ask your agent system
If the implementer changed just the producer field in its own evaluation receipt to the expected reviewer name, what independent control would prevent that receipt from authorizing the next step?
I am particularly interested in minimal examples that prove the boundary without adding an entire second orchestration platform.
Disclosure: This is a synthetic fixture and a proposed evidence boundary, not a claim that ARC currently enforces runtime producer authentication. The runnable .mjs file accompanies the draft and should be published at a stable source URL before referring to it as publicly reproducible.
Top comments (9)
The one-field problem is real, and the field to distrust is claimedProducer itself: while the checker reads it, an implementer can write it. The fix is not a stricter comparison, it is evidence the implementer cannot produce - an HMAC over (candidate, verdict) with a key scoped to the reviewer identity, so forging requires a key rather than a string.
There is also a cheap test for whether that field ever mattered: delete claimedProducer from a legitimate event and re-run. If the verdict stays PASS, the field was decorative and can go. If it flips to FAIL, the field was load-bearing, and so is the forgery.
That pair - absent field, present forgery - is what makes the fixture about authority instead of labels.
That's a useful distinction, 于侃. The missing-field test tells us whether a label is load-bearing; the forgery test tells us whether the verifier mistakes that label for authority.
I agree that the HMAC key must be controlled by an independent reviewer trust boundary, not accessible to the implementer.
I'd also bind the authenticated receipt to the exact candidate SHA, verdict, review policy version, and run identity. Otherwise, a legitimate PASS receipt for commit A could be replayed against commit B.
That suggests a third negative fixture: approve A, change the candidate to B, replay A's valid receipt, and require the merge gate to reject it.
One further distinction: a valid HMAC establishes receipt provenance, but doesn't by itself prove that an independent review actually took place.
Would you include stale-SHA replay alongside the missing-field and forged-field cases?
Both, but the split matters more than the frequency. Three recheck triggers, not one clock. TTL-based blockers expire on their own and want a cron. State-based ones - a dependency version, a feature flag - only change when something writes, so they want a write-trigger rather than a timer. Authority-based ones - a denied permission, a missing approval - change only by an authorized act, and a timer there is actively harmful, because it turns a refusal into a re-ask loop.
The failure I see most is one retry clock for all three. It converts a state blocker into "wait and try again" and an authority blocker into nagging, and it produces a BLOCKED list nobody trusts, because entries expire for no reason.
Exactly. The distinction is what is allowed to wake the task, not just how often it is retried.
I'd model three independent wake-up paths:
One edge case: state changes outside the scheduler's write boundary, or lost change events. I'd add bounded, read-only reconciliation for state blockers without turning it into a generic execution retry.
I'd test the boundary with three negative cases: expired TTL while STOP/HOLD remains active; an unobserved dependency change; and a missing approval that never prompts again until an authorized change.
This keeps "ready to re-evaluate" distinct from "authorized to execute."
Does your state-based path rely exclusively on change events, or do you also have a reconciliation fallback?
Your stale-SHA replay question is exactly where this bites for me. Old verdicts expire in my setup. A 'blocked' or 'impossible' note from three weeks ago isn't treated as fact — it's a measurement from that day, and if the environment changed, it gets re-measured before anyone stops work on it. Stale verdicts quietly killed more momentum for me than real blockers did.
That's a useful distinction. A PASS receipt can only support the exact candidate and policy it evaluated; a BLOCKED verdict is likewise evidence of conditions at a particular time, not a permanent property of a task.
I'd bind it to the candidate SHA, observed_at, environment fingerprint, blocker class, and a recheck trigger.
A changed dependency or expired TTL should trigger a fresh measurement, never silently convert BLOCKED into permission to execute.
I'd keep authority gates separate: STOP/HOLD, denied permissions, and required approvals remain in force until an authorized change.
In your setup, what most often invalidates a blocker—time, a specific environment change, or both?
Policy version is the field I would think hardest about, because it decides whether a receipt survives a policy edit. Checked at verification time, a policy tweak silently invalidates every open PR - correct, but traumatic. Recorded only for audit, the version is decoration, which is the failure the missing-field test exists to catch. The honest middle is: version binds only for receipts issued before the change, with an explicit grandfathering window.
Run identity has the same shape as your replay point. If the run id is minted at the end of the review, it is provenance written after the fact. It has to come from a scheduler that existed before the reviewer started, so the receipt references a value the reviewer could not have created.
Agreed on both trust boundaries.
I'd bind each receipt to the exact candidate SHA and a policy content digest, then have the verifier consult an authorized, immutable transition record defining which older policies remain valid, for which scope, and until when.
Grandfathering should be explicit and limited to compatible changes; security-critical revisions or revocations should invalidate old receipts immediately.
For run identity, the scheduler should issue the run ID before review begins and bind it to the authorized input and reviewer principal. The implementer must not be able to mint or rewrite that record or use the reviewer's signing authority.
I'd add negative tests for an expired grace window, an emergency policy revocation, and a run ID minted after the verdict.
Would you require an explicitly approved compatibility decision for each grandfathering window?
Yes, reconciliation has to exist - events get lost, and a blocker nobody re-reads is a blocker forever. The constraint I would hold it to is that reconciliation may only emit evidence, never perform the BLOCKED-to-READY transition itself; otherwise "read-only" is a naming exercise and you have rebuilt the retry with more steps.
Two bounds make that real. A wall-clock floor, because lost events are rare and reconciling every minute is just a slower poll. And an emit ceiling per blockage: if the same unchanged state keeps being re-emitted, that is a lost-event storm and it should page a human rather than keep reconciling.
Your third negative case is the sharpest test: if reconciliation ever re-notifies an approver, it has become the re-ask loop wearing a different name.