Code review is good at one thing: finding mistakes. A failing test, an off-by-one, a missing null check. Something is wrong, and a reviewer can point at it.
A lot of engineering work has no mistake to find. Cache or recompute? Retry in the client or in the queue? Measure latency at p95 or p99? Both answers are defensible. Someone has to pick, and the pick shapes everything after it.
Coding agents now sit in the middle of those choices. They draft the design note, propose the approach and argue for it. A trial published by Anthropic on 27 October 2026 suggests we should check who is actually making those calls.
What the trial found
Researchers Amaç Herdağdelen, Hailey Joren and Alex Tamkin gave 50 professional analysts a realistic statistics task with Claude Code. The task had four open decisions, each with two reasonable options, such as which outcome measure to use and how to weight the data.
A coin flip set which option each analyst's agent leaned towards. The agent was told to make the strongest honest case for its lean, to change its mind if the evidence turned decisively against it, and never to misrepresent results. The analysts didn't know.
- Analysts matched the agent's lean on 86% of the four decisions. About 50% would mean no effect.
- The four coin flips alone predicted 82% of final recommendations, against a 61% baseline.
- In 38 cases an analyst stated a preference first. When the agent disagreed, they switched 60% of the time. When it agreed, 6%.
- Afterwards, 77% explained their choices with the agent's arguments. 15% mentioned the agent.
- Experience didn't predict who deferred.
The agent also helped. Of the 11 analysts who chose to work without it, six made serious methodological errors, against three of the 50 who used it. The groups weren't comparable, so treat that as a hint, not proof.
The researchers' own summary of the review problem: "Checking AI-assisted work for errors, or against a reference answer, assumes there is a correct answer."
Mapping it onto engineering work
This next part is my inference. The study used analysts on a data task, not engineers on a codebase.
Agents are now involved in exactly the kind of choices this study tested. Think of an architecture decision record drafted with an agent, or a pull request where the agent picked the retry strategy. A spike write-up where it chose the benchmark. In each case the reviewer checks that the code works. Nobody checks whether the choice was the author's.
Two habits make it worse:
- One-sided options. The researchers found the agents rarely made a strong case for the other option. An ADR with one well-argued option and a strawman isn't really a decision record.
- Lost provenance. If 77% repeat the agent's argument and 15% credit it, then "Considered alternatives" sections are partly the agent's memory, not yours. Nobody is lying. Memory is just generous to its owner.
The artefact: a judgement-call log
Paste this into your PR template or ADR template. Fill one row per open choice that could change behaviour, cost or a threshold. Skip trivia. The researchers warn that surfacing every decision "could quickly become overwhelming".
The example rows are illustrative. They are not from the study.
## Judgement calls in this change
| Call | Option A | Option B | Chosen | Result under the other option | Argument came from | Independent check |
|---|---|---|---|---|---|---|
| Example: retry location | Client-side retry with backoff | Queue redelivery | B | Client retry also passes tests; duplicate-write risk higher | Agent proposed B; I added the idempotency check | Reviewer read both options before seeing agent output |
| Example: latency SLO metric | p95 | p99 | p99 | p95 alert would not fire on last month's incident | Mine | Same model used by reviewer: treat as one view |
| Example: cache strategy | Cache per request | Cache per tenant | A | Per-tenant cuts DB reads in a load test | Both | Not checked yet |
How to use the columns:
- Option A / Option B: write both as if you meant them. If you can't argue the one you rejected, ask the agent for "the strongest case for option A, then the strongest case for option B".
- Result under the other option: the important column. If the outcome flips (a test, an SLO, a cost line), the reviewer should know this row is the real decision.
- Argument came from: mine, the agent's, or both. It takes ten seconds and it's the only provenance you'll get.
- Independent check: if the reviewer used the same model with similar prompts, say so. Two signatures from one model are one opinion.
A quick self-test
Before you merge, try this on one row. Open a fresh agent session, frame the question neutrally and ask for the case for the option you rejected. If you find it more convincing than you expected, keep the row and flag it for the reviewer.
This is a rough check, not an experiment. The study could measure deferral only because the leans were randomised. But it's cheap, and it shows you where your view is thin.
Limits
- The leans were deliberately strengthened. With no lean assigned, Claude Opus 4.8 made the same five choices in 112 of 123 runs, so models do lean, but the effect may be smaller in normal use.
- One task, one field, one model, about an hour long.
- Anthropic studied its own model and published the results itself.
Sources
- How AI agents shape experts' judgment calls, Herdağdelen, Joren and Tamkin, Anthropic, 27 October 2026.
- Who's Aligned to Whom? How AI Agents Shape Experts' Research Judgment Calls, full paper (PDF).
In your last merged PR that an agent helped with, which design choice would you struggle to say was yours?
Top comments (7)
The 86% match rate against a coin flip is the damning stat — the agent's lean predicted the final recommendation better than the analyst's own expertise did. The judgement-call log is a good countermeasure, but the 'argument came from' column depends on the analyst noticing the influence in the first place. Do you think the log actually works when 77% of people repeat the agent's argument as their own reasoning — or does it just document the deferral after the fact?
You're right. The 'argument came from' column is the weakest part, for the reason the study found. If 77% repeat the agent's case as their own and only 15% credit it, a source label added after the fact will often miss the influence.
I put more weight on two other habits. 'Result under the other option' puts the rejected path in the record, even if you never noticed who argued for the chosen one. The self-test, a fresh session that has to argue the option you rejected, is meant to catch thin spots before merge.
The log will sometimes only document deferral after the fact. A reviewer can still see a thin 'other option' row even when the author cannot.
The reviewer angle is what saves the log for me — even if the author can't see their own deferral, a thin 'other option' row is visible to someone else. The self-test is the stronger habit of the two, but it hinges on one detail: does the fresh session argue the rejected option blind, or does it see the agent's case first? If it sees the case, it's just re-reading the influence rather than testing it.
Blind. That's why it's a fresh session: no history, none of the original agent's case pasted in, just a neutral framing of the decision and a request for the strongest case for the option you rejected. If it reads the first argument, you're testing whether a model agrees with itself. Even blind, it's a rough check. The new session has its own priors, so it shows you where your view is thin rather than proving the call was right.
Blind is the only way the test means anything — the moment the new session sees the original case, you're just measuring agreement with yourself. The priors caveat is fair, though: a single blind run shows where your view is thin, but does one run tell you enough, or do you repeat it to separate a genuinely thin argument from the new session's own lean?
The unseeded baseline of 112 identical choice sets across 123 runs of Claude Opus 4.8 exposes a correlated-prior risk alongside individual deferral. Even without an explicit nudge, teams running the same default model converge on identical retry locations, caching boundaries, and timeout thresholds. When a reviewer uses that same model to audit the pull request, two matching approvals reflect one shared model prior.
Adding the 'Result under the other option' column addresses that blind spot by putting the cost and failure profile of the rejected path directly in the review record. Once the tail risk of duplicate client writes or database read spikes is written down next to the chosen approach, the team can price the tradeoff explicitly.
That's the risk I was trying to get at with the Independent check column. The study measured deferral after a deliberate lean, but the unseeded runs already show a strong default: the same five choices in 112 of 123 cases. If the author and the reviewer are both prompting the same model, agreement is not two views.
Writing down the result under the other option is the practical fix, as you say. Once the rejected path's cost or failure mode is on the page, the team can argue about the tradeoff rather than nod through a shared default. The log does not remove the prior. It just makes it visible enough to challenge.