The two replies that have cost me the most time are short ones. "It's fixed." And "that isn't used anywhere." Both sound like a place you can stop. I would read one, move on to the next thing, and only find out later that nothing had run since the last edit, or that the agent had searched once and decided the question was settled.
I run a one-person company on long-running Claude Code sessions and scheduled AI jobs. When an agent tells me a bug is gone and I believe it, I move on. The mistake usually shows up after I've already started something else.
So I wrote a Stop hook. When the agent tries to end its turn after a claim like "fixed," "tests pass," "deployed," or "doesn't exist," the hook looks at what actually happened in the session. If the proof isn't there, it sends the agent back once and says what's missing. The note looks like this:
claim-check: you wrote "is fixed", but nothing ran successfully after the last edit.
Run the check that proves it (the tests, the script, a request to the live URL) and
show the result, or say plainly that it isn't verified yet.
It never blocks twice in a row, so the agent can't get stuck. If the hook itself crashes, it lets the turn end. And if the agent just says the work isn't verified yet, that always passes. I wanted a way to make it show its work, not a way to trap it.
On November 4, 2026, I ran the hook over my own Claude Code history to see what it would have done. After dropping 1,649 duplicate turns (resuming a session copies earlier turns into a new file), I had 3,977 turns left. 406 of them ended with a claim like "done," "fixed," "deployed," or "tests pass."
The version I measured this way, 0.2.0, would have blocked 86 of those 406. Most of the 86 were not the case I had built it for.
77 had no tool calls at all. Those were usually wrap-ups: the agent saying something was fixed or deployed, with no commit, test result, or status code in the reply. The proof may have been in an earlier turn, but the wrap-up is what I actually read, and it didn't show any. 8 had tool calls, but only reads, searches, read-only commands, to-do updates, or failed commands. 1 made the claim after a code edit with nothing run successfully since. That last one is the case I cared about. It's probably rare in my numbers because an older version of the hook was already catching it on my machine. Your count may be higher.
I hand-checked 24 of the 86: 20 drawn at random when the count was still 82, plus 4 that later rule changes added. Eighteen were wrap-ups that claimed finished work without citing a commit, test output, or status code. Three weren't claims of finished work at all: a research finding (a change had been tested and made the model worse), a condition (the work would count as done once merged), and a plan for work that hadn't started. The other three could be read either way.
After I posted those numbers, someone argued that the 77 wrap-ups were the ones that mattered, because a hook that blocks a correct wrap-up gets turned off. They had a point. In 76 of those 77, something had run successfully after the last edit, just in an earlier turn. So 0.3.0 started counting a check from earlier in the session, as long as no code was edited after it. Of the same 406 claims, it now blocks 2: the one after an edit with nothing run, and one wrap-up with nothing run since the last edit.
The tradeoff is real. A wrap-up no longer has to show its proof, so the wrap-ups I hand-checked get through now too. I'd rather catch the dangerous case and stay installed than win the argument about tidy endings.
Separately, 17 turns ended by saying something wasn't there, and in 2 of them the agent had looked once or not at all. That's the other reply that has burned me. For a "doesn't exist" or "nothing calls it" claim, the hook wants two lookups in that turn, of different kinds. Searching for the same text twice doesn't count, even in another folder or with different flags. A different text, a different file, or a different tool does.
For a "fixed" or "tests pass" claim, what it wants is simpler. Something has to have run after the last code edit and not failed: a shell command, an MCP tool, a URL fetch, or a subagent. git diff and cat don't count, and neither does a command that failed. If the turn didn't edit code, a successful check from earlier in the session is enough, as long as no code was edited after it. The hook will also accept proof cited in the reply itself: a commit hash, a line like "14 passed" with no failures, a 2xx status, "exit code 0," or a link to a CI run. "3 failed, 2 passed" does not count, and neither does "0 passed."
A few of the gaps are worth saying out loud, because they are how you end up turning a hook off.
It matches set phrases. "The redirect loop is fixed" and "Fixed." get caught. "Fixed the redirect loop," "Fixed, deployed to prod," and "I'm done" do not. A sentence that opens with a condition, like "After the restart, everything is deployed," gets through. One honest "not verified" anywhere lets the whole reply through, including an unrelated one like "Fixed. Do not test on prod." It only understands claims in English and Chinese. And a command after the edit only proves that something ran. It doesn't prove the command tested the right thing.
If you want to see what it would do on your own history before it starts sending the agent back, clone the repo and run python3 claim_check.py replay. It reads past sessions in ~/.claude/projects and prints how many claims it would have blocked. Nothing leaves your machine. You can also set the plugin's log_only option to true, leave it for a week, and read the log before you turn blocking on.
For Claude Code, the install is:
/plugin marketplace add Balance0014/claude-code-claim-check
/plugin install claim-check@claim-check
Then reload plugins if Claude Code asks, or start a new session. It needs Python 3.9+ and has no dependencies. If you don't use the plugin system, you can point a Stop hook in ~/.claude/settings.json at claim_check.py. For Codex and other agents, npx skills add Balance0014/claude-code-claim-check installs the rules as a skill. The agent can read them, but nothing makes it follow them. There's no hook for Codex in this release.
The hook and the replay script are in claim-check. In the week before I published the first version, the older copy on my machine blocked 19 times. That was enough to keep it on.
If the thing eating your week isn't a false "fixed" but the way long conversations get re-read on every reply, I also made when-to-clear. It reads the token counts Claude Code already saves locally, tells you when it's time to /clear, and can write a handoff note so a new session can pick up the work. Different problem, same machine.
Top comments (19)
For the exit-0 gap, the check that has worked best for me costs one extra run: take the edit back out, run the same command, and it has to fail. If it still passes without the fix, it never exercised the fix, whatever the exit code said. Then put the edit back. It doesn't need the transcript at all, and the agent can do it itself before it says "fixed".
The claim that bit me yesterday was "released". The PR was green, the merge was green, and the publish job that runs after the merge had failed on an unrelated flaky test, so the registry still served the previous version. A link to the CI run would have passed your hook, and for that claim it proves the wrong job. What settled it was asking the registry which version it serves. For "deployed", the proof lives at the destination, not in the pipeline.
Thanks, both of these are better than what the hook does today. The "take the edit back out and it has to fail" check is the one I keep wishing I could ask the agent to do by default. My worry is slow test suites, where a second run costs real time, so I'd probably only ask for it when the claim is about a bug fix.
The "released" example is a good catch. You're right that a green CI link proves the wrong job there. The hook only checks that something ran, not that it looked at the destination, so asking the registry which version it serves is a gap I can't close with a transcript check alone.
The set-phrase gap is the one I'd worry about, because an agent can say "done" a hundred ways. In our pipeline at Oizom we gate the action instead of the words: the agent ends its work by calling endWork, and that call is refused until its plan is finished, its stack is healthy and the merge is clean. Your replay over 406 real claims is the part I'd copy. Did the 77 wrap-ups cluster at the end of long sessions, or spread evenly?
I didn't split the 77 by session length, so I can't say yet. I only tagged them as wrap-ups where the check had run in an earlier turn. That's a useful next cut, especially to see whether longer sessions are where the stale "done" signal shows up.
Fair, that cut can wait. From the same replay I'd want to know about the 2 that v0.3.0 still blocks: were they real cases where nothing ran, or false alarms on a phrase? Two out of 406 is small enough to read by hand.
Accepting "a shell command, an MCP tool, a URL fetch, or a subagent" as the thing that ran leaves one path unchecked. Per the hooks docs, Stop runs when the main agent finishes responding, and a subagent finishing fires SubagentStop instead. The plugin's hooks.json registers only Stop. So if the main agent delegates the test run and the subagent reports "all tests pass" without running them, nothing checks that reply, and the Agent call then counts as the successful run for the main turn's claim.
SubagentStop looks like a fit for the same script. Its input carries
last_assistant_messagewith the subagent's final text, it accepts the same block decision as Stop, and the block reason is delivered to the subagent as its next instruction. Two details in that section need handling.transcript_pathis still the main session's transcript there, and the subagent's own file is inagent_transcript_path. The event also fires for internal agents such as prompt suggestions, whereagent_typeis the session's agent name or an empty string, and a matcher that names agent types skips the empty ones.If Stop only runs when the main agent finishes, then a delegated test run would sneak through exactly the way you described. The Agent call itself can look like a successful run, and the subagent's "all tests pass" never gets checked.
I'll look at adding SubagentStop with the same script, and I'll be careful about the transcript_path vs agent_transcript_path distinction you flagged. The empty agent_type case is easy to miss too; a matcher that only lists named types would skip exactly the internal ones I wouldn't want to ignore, or worse, trip on them. I'll need to handle that explicitly rather than assuming a type list is enough.
Thanks for laying this out so clearly. This is the kind of hole that would have stayed invisible in my own replay set, because most of those turns weren't delegated.
The drop from 86 blocks down to 2 out of 406 claims explains why most verification controls get turned off. In v0.2.0, roughly 98% of the blocks interrupted valid wrap-ups where a check had already run earlier in the session, which creates enough friction that people uninstall the hook inside a week.
Accepting an earlier-turn check when no code changed afterward trades tidy wrap-up formatting for keeping the guardrail enabled. The remaining gap you pointed out is that an exit code of zero only shows a command finished, leaving open whether it exercised the edited branch. Trying to prove full test coverage from transcript replay hits diminishing returns fast, so blocking the silent no-run claim right after an edit captures most of the downside without bogging down normal turns.
Yeah, that remaining gap is the one I'm least happy about. Exit 0 only means a command finished, not that it touched the branch I just edited. I poked at inferring that from the transcript and it fell apart pretty quickly, too many ways a test command can look.
So I stuck with blocking the silent no-run right after an edit. That's the case that actually wasted my time, and it's cheap enough that I leave the hook on. A later check that the command even mentioned the edited files might be worth it, but I'd still leave real coverage to CI.
The rule that an honest "not verified yet" always passes is the part I like most. I came to something similar from the other side: in my setup a check that couldn't give an answer, like a timeout or a diff it couldn't read, is a separate state, "don't know", not a pass and not a fail. Before that, "done", "not checked" and "the process just died" all showed the same green badge. About the gap you're least happy with, exit 0 not proving the command touched the edited code: I don't think the transcript can answer that either. What helped me was moving the question one level up. A separate read-only agent checks each phase against the plan, and the tests are written from the plan, not by the agent after its own code. It doesn't prove coverage, but at least someone asks whether the right thing was tested.
That distinction between “don’t know” and pass or fail is useful, especially when a timeout or unreadable diff makes the evidence incomplete. My Stop hook is deliberately narrower: it catches a confident “it’s fixed” claim when I can’t find proof in the session, but an exit code still can’t tell me whether the right code was tested. I like the read-only plan check as a separate layer rather than making the hook pretend it can prove coverage.
The mixed-status escape hatch seems worth a replay fixture: "The redirect is fixed. Deployment isn't verified yet" contains a verified-sounding claim and an honest caveat about a different task. If the caveat clears the whole reply, the first claim never needs evidence. Would scoping "not verified" to the specific claim sentence catch this while keeping your one-block limit and low false-positive rate?
That's a good fixture to add. Right now a "not verified" anywhere in the reply can clear the whole thing, even when it's sitting next to a confident claim about something else. I'm leaning toward scoping the caveat to the claim it belongs to. The case I want to be careful with is "the redirect is fixed. I'll verify the deploy next" — the second sentence is a plan, and I don't want that to force a block on a first claim that was already checked. I'll run it through a replay and see where it breaks.
the 77 wrap ups with no tool calls is the bit that stuck with me, the summary is the only thing you actually read so if the proof isnt in it it may as well not exist. same worry with travel answers in searchspot, a confident line with nothing checked behind it is the dangerous one
That's the part I didn't expect either. The summary is the only thing I actually read, so if the proof isn't in it, I treat it as unverified. Now I look for the command and its result in the same message before I believe it. Thanks for reading it that closely.
@gapmine agreed, and that is the gap to close. A 'fixed' that arrives with the command and the last lines it printed can be checked; without the tail, '0 tests collected' passes the same way a real run does. Requiring that output tail in the claim closes exactly that hole.
Agreed. A clean exit can still hide a run that collected zero tests. I'd want the receipt to include the test count and the last output too, not just the command and exit code. That's much harder to mistake for a real pass.
A 'fixed' is only checkable when it arrives with the command that ran and the last lines it printed. Without those, the claim is the agent grading its own work.
That's a clean way to put it, thanks Paul. The part I'm still unsure about is the printed lines. My check only looks at whether a command ran after the last edit, so a run that printed "0 tests collected" looks the same as a real pass. Requiring the output tail in the claim would catch that one.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.