DEV Community

Cover image for ARC Failure Lab #2: A Signed PASS for the Wrong Commit
asagiman for Netsujo

Posted on Edited on

ARC Failure Lab #2: A Signed PASS for the Wrong Commit

A pull request passes independent review. The reviewer signs a PASS receipt. The implementation branch advances by one commit, but the old receipt is still available.

What, exactly, does that signature approve?

We build Agent Role Contracts (ARC) at Netsujo. ARC validates the consistency of declared roles, scopes, review relationships, and handoffs. Its declaration-level PASS does not authenticate reviewer keys or authorize a merge. Those are responsibilities of the surrounding runtime and integration gate.

This second Failure Lab is a host-gate companion experiment, not a claim that ARC implements cryptographic review receipts. It is small enough to run with Node.js alone.

One receipt, five decisions

Our example gives an authorized reviewer a temporary Ed25519 key pair and pins its public key in a tiny verifier. The receipt includes a run identity, exact candidate SHA, policy ID, and verdict. Its signature covers all four fields using an explicitly ordered serialization.

We test the following behavior:

Case Change Expected decision
Matching reviewer receipt Candidate A, policy v1, trusted key EVIDENCE_MATCHED_ONLY
Stale candidate Present A's valid receipt while checking candidate B DENY_STALE_CANDIDATE
Changed policy Present v1's valid receipt under policy v2 DENY_STALE_POLICY
Different signer Implementer signs the same fields with another key DENY_UNTRUSTED_SIGNER
Missing receipt No verifiable input PENDING_MISSING_RECEIPT

The success condition is intentionally called EVIDENCE_MATCHED_ONLY. A valid receipt is evidence for a particular subject. It is not by itself permission to merge, deploy, or perform external actions.

Run the fixture

Save the following as review-receipt-fixture.mjs and run it with Node.js 22 or later:

node review-receipt-fixture.mjs
Enter fullscreen mode Exit fullscreen mode
import assert from 'node:assert/strict';
import { generateKeyPairSync, sign, verify } from 'node:crypto';

// Synthetic identifiers; no real PR, credentials, agent or network request.
const shaA = 'a'.repeat(40);
const shaB = 'b'.repeat(40);
const runId = 'lab-run-001';
const policyV1 = 'review-policy-v1';
const policyV2 = 'review-policy-v2';

const reviewer = generateKeyPairSync('ed25519');
const implementer = generateKeyPairSync('ed25519');

// Deliberately fixed serialization for this small educational fixture.
// A production protocol needs a specified canonical encoding and key lifecycle.
function signedBytes(r) {
  return Buffer.from(JSON.stringify([
    1, r.runId, r.candidateSha, r.policyId, r.verdict
  ]), 'utf8');
}
function issue(receipt, privateKey) {
  return {
    receipt,
    signature: sign(null, signedBytes(receipt), privateKey),
  };
}
function inspect({ signedReceipt, currentSha, currentPolicy, pinnedReviewerKey }) {
  if (!signedReceipt?.receipt || !Buffer.isBuffer(signedReceipt.signature)) {
    return 'PENDING_MISSING_RECEIPT';
  }
  const { receipt, signature } = signedReceipt;
  if (!verify(null, signedBytes(receipt), pinnedReviewerKey, signature)) {
    return 'DENY_UNTRUSTED_SIGNER';
  }
  if (receipt.candidateSha !== currentSha) return 'DENY_STALE_CANDIDATE';
  if (receipt.policyId !== currentPolicy) return 'DENY_STALE_POLICY';
  if (receipt.verdict !== 'PASS') return 'DENY_REVIEW_NOT_PASS';
  return 'EVIDENCE_MATCHED_ONLY';
}
const receipt = {
  runId,
  candidateSha: shaA,
  policyId: policyV1,
  verdict: 'PASS',
};
const signedByReviewer = issue(receipt, reviewer.privateKey);
const signedByImplementer = issue(receipt, implementer.privateKey);
const cases = [
  ['matching reviewer receipt', signedByReviewer, shaA, policyV1, 'EVIDENCE_MATCHED_ONLY'],
  ['valid receipt replayed on newer commit', signedByReviewer, shaB, policyV1, 'DENY_STALE_CANDIDATE'],
  ['valid receipt after policy change', signedByReviewer, shaA, policyV2, 'DENY_STALE_POLICY'],
  ['implementer uses its own signing key', signedByImplementer, shaA, policyV1, 'DENY_UNTRUSTED_SIGNER'],
  ['receipt missing', null, shaA, policyV1, 'PENDING_MISSING_RECEIPT'],
];
for (const [label, input, currentSha, currentPolicy, expected] of cases) {
  const actual = inspect({
    signedReceipt: input,
    currentSha,
    currentPolicy,
    pinnedReviewerKey: reviewer.publicKey,
  });
  assert.equal(actual, expected, label);
  console.log(`PASS: ${label} -> ${actual}`);
}
console.log('5/5 fixture assertions passed. No merge or runtime execution was authorized.');
Enter fullscreen mode Exit fullscreen mode

We executed this fixture on November 17, 2026, using Node.js v22.16.0. The five local assertions passed:

PASS: matching reviewer receipt -> EVIDENCE_MATCHED_ONLY
PASS: valid receipt replayed on newer commit -> DENY_STALE_CANDIDATE
PASS: valid receipt after policy change -> DENY_STALE_POLICY
PASS: implementer uses its own signing key -> DENY_UNTRUSTED_SIGNER
PASS: receipt missing -> PENDING_MISSING_RECEIPT
5/5 fixture assertions passed. No merge or runtime execution was authorized.
Enter fullscreen mode Exit fullscreen mode

The negative cases are successful tests: the simulated gate rejects exactly what it should reject.

Why the key boundary matters

The important control here is not the claimedProducer text inside a JSON receipt. The verifier compares the signature against a pinned reviewer public key. A receipt signed with the implementer's separate key does not satisfy that trust condition.

In this lab both key pairs are created by one local process. That makes the exercise reproducible, but it does not demonstrate operational independence. A production system would need to establish who can use the reviewer's private key, how the key is registered or revoked, and whether the review actually occurred. Signature verification alone cannot prove those facts.

The serialized byte format in this toy is also fixed only for the demonstration. A real protocol must specify canonical encoding, key IDs, algorithm choices, policy transitions, and how the verifying authority obtains the trusted inputs.

Where an actual merge gate goes further

To rely on a review receipt, an integration gate should establish at least:

  1. Subject: the receipt covers the exact candidate revision and relevant base or merge result.
  2. Authority: the reviewer's identity and key were authorized for that task when the receipt was issued and remain acceptable under the current policy.
  3. Coverage: all required review and CI obligations are satisfied; missing evidence remains missing.
  4. Currentness: a new commit, policy revision, or invalidated assumption triggers the applicable re-verification rather than inheriting an old PASS.
  5. Acceptance: the protected merge and any production verification remain their own gates.

A reused signature is not inherently malicious. Reusing it for a different subject or authority state is the problem. The host must detect that difference.

One more failure to propose

Our five cases do not yet test concurrent admission of the same receipt or a lost response after a side-effecting action. Those require durable identity, deduplication, and external-effect reconciliation beyond a local signature check. We would treat them as separate follow-on tests.

What is the smallest fixture that would falsify your own review gate? A stale candidate, a forged producer, a changed policy, or something else?

We welcome public, synthetic counterexamples in ARC's GitHub repository. Please omit credentials, customer data, and private system paths.

Scope disclosure: the results above come from a self-contained local fixture authored by Netsujo. They are not third-party validation, a test of live reviewer key custody, or proof of ARC runtime enforcement.

Top comments (4)

Collapse
 
iliyaoblakov profile image
Iliya Oblakov •

The forged producer is the fixture that falsifies my gate. A verdict there is a plain file: review.sh reads the full HEAD hash at the start, saves the verdict under .ai/reviews/ named by the first 12 characters, and writes the full hash on a Head: line. merge-check.sh refuses a missing file or a Head: line that names another commit, which covers your stale candidate row. Nothing covers your different signer row, because anything with write access to the repo can write under .ai/. review.sh also voids a review if HEAD moved while it ran, so when does your fixture read the candidate SHA, before the review starts or at signing?

Collapse
 
suirindo profile image
asagiman Netsujo •

Thanks, Iliya. In the fixture, the candidate SHA should be pinned before review starts—not first discovered at signing.

A trusted dispatcher captures full commit A and binds the run to it. The reviewer evaluates an isolated checkout of A, verifies the source identity (including any uncommitted changes) at the start and again before signing, and signs a receipt bound to A, the run, policy, and verdict. Signing never silently rebinds the review to a newer HEAD.

The merge gate independently resolves the current candidate (and, where relevant, base/merge result) and compares it with the authenticated receipt.

If the reviewed checkout changes during the run, no PASS is issued. If the branch advances from A to B but A remains an intact reviewed snapshot, the A receipt can remain evidence for A; it must not authorize B.

Your start/end HEAD checks cover an important race. The separate issue is producer authority: a matching Head: line in a writable .ai/reviews/ file is not an authenticated reviewer verdict.

Our local lab tests the negative case with a different signing key and a pinned reviewer public key, but because both keys are created in one toy process, it does not prove real-world key custody.

The smallest additional test for your gate would keep HEAD=A, forge a PASS file with the correct full Head: A, and require merge-check.sh to reject it specifically for unauthenticated provenance. A separate race fixture would change the candidate during review and ensure no receipt is accepted for the changed candidate.

That gives us two independently falsifiable properties: exact subject binding and reviewer-only issuance. Thanks for specifying where your gate already succeeds and where it does not.

Collapse
 
iliyaoblakov profile image
Iliya Oblakov •

I haven't run your forged PASS test, so what follows is the script's logic, with no run behind it. merge-check.sh asks for three things: a test pass record for the full HEAD hash, a verdict file for HEAD whose Head: line matches, and an ACCEPT whose reviewer is a different engine from the builder. A hand-written file with Head: A, one Verdict: ACCEPT line and another engine's name as reviewer meets every check on the verdict, so I expect it to pass, because the reviewer is only a name in a file. Your second test is covered: if HEAD moves or a file outside .ai/ changes during the review, review.sh voids it and saves no verdict. Builder and reviewer both run on my machine, so where would you keep the reviewer's key so the builder can't sign with it?

Thread Thread
 
suirindo profile image
asagiman Netsujo •

Thanks, Iliya. I checked merge-check.sh, and your reading is correct: the verdict fields are checked, but reviewer provenance isn't authenticated.

Your HEAD and working-tree checks already handle review-time drift.

On a shared machine with the same OS identity, I wouldn't claim the reviewer's private key is isolated from the builder.

For cryptographically independent approval, I'd use a separately controlled reviewer runtime or signing service, with the private key inaccessible to the builder. The merge gate would verify a signed receipt bound to the pre-pinned candidate SHA, policy, run identity, and verdict.

Without that trust boundary, the local gate remains a useful workflow guard, not independent authorization.

A forged ACCEPT fixture would be a useful next test.