Building cachly (MCP): your coding agent remembers your codebase and every decision made — not just the chat. Follows you between editors, stays when someone leaves. I post what I learn shipping it.
For the exit-0 gap, the check that has worked best for me costs one extra run: take the edit back out, run the same command, and it has to fail. If it still passes without the fix, it never exercised the fix, whatever the exit code said. Then put the edit back. It doesn't need the transcript at all, and the agent can do it itself before it says "fixed".
The claim that bit me yesterday was "released". The PR was green, the merge was green, and the publish job that runs after the merge had failed on an unrelated flaky test, so the registry still served the previous version. A link to the CI run would have passed your hook, and for that claim it proves the wrong job. What settled it was asking the registry which version it serves. For "deployed", the proof lives at the destination, not in the pipeline.
Built a crypto media site and an on-chain analytics tool. Now I invest in crypto and follow post-quantum security. Nights I vibe code and dig Reddit for startup ideas.
Thanks, both of these are better than what the hook does today. The "take the edit back out and it has to fail" check is the one I keep wishing I could ask the agent to do by default. My worry is slow test suites, where a second run costs real time, so I'd probably only ask for it when the claim is about a bug fix.
The "released" example is a good catch. You're right that a green CI link proves the wrong job there. The hook only checks that something ran, not that it looked at the destination, so asking the registry which version it serves is a gap I can't close with a transcript check alone.
The set-phrase gap is the one I'd worry about, because an agent can say "done" a hundred ways. In our pipeline at Oizom we gate the action instead of the words: the agent ends its work by calling endWork, and that call is refused until its plan is finished, its stack is healthy and the merge is clean. Your replay over 406 real claims is the part I'd copy. Did the 77 wrap-ups cluster at the end of long sessions, or spread evenly?
Built a crypto media site and an on-chain analytics tool. Now I invest in crypto and follow post-quantum security. Nights I vibe code and dig Reddit for startup ideas.
I didn't split the 77 by session length, so I can't say yet. I only tagged them as wrap-ups where the check had run in an earlier turn. That's a useful next cut, especially to see whether longer sessions are where the stale "done" signal shows up.
Fair, that cut can wait. From the same replay I'd want to know about the 2 that v0.3.0 still blocks: were they real cases where nothing ran, or false alarms on a phrase? Two out of 406 is small enough to read by hand.
Independent directory of the Claude Code skills, MCP servers & marketplaces worth installing. 86,000+ tools indexed daily, ranked by real installs. Not affiliated with Anthropic.
Accepting "a shell command, an MCP tool, a URL fetch, or a subagent" as the thing that ran leaves one path unchecked. Per the hooks docs, Stop runs when the main agent finishes responding, and a subagent finishing fires SubagentStop instead. The plugin's hooks.json registers only Stop. So if the main agent delegates the test run and the subagent reports "all tests pass" without running them, nothing checks that reply, and the Agent call then counts as the successful run for the main turn's claim.
SubagentStop looks like a fit for the same script. Its input carries last_assistant_message with the subagent's final text, it accepts the same block decision as Stop, and the block reason is delivered to the subagent as its next instruction. Two details in that section need handling. transcript_path is still the main session's transcript there, and the subagent's own file is in agent_transcript_path. The event also fires for internal agents such as prompt suggestions, where agent_type is the session's agent name or an empty string, and a matcher that names agent types skips the empty ones.
Built a crypto media site and an on-chain analytics tool. Now I invest in crypto and follow post-quantum security. Nights I vibe code and dig Reddit for startup ideas.
If Stop only runs when the main agent finishes, then a delegated test run would sneak through exactly the way you described. The Agent call itself can look like a successful run, and the subagent's "all tests pass" never gets checked.
I'll look at adding SubagentStop with the same script, and I'll be careful about the transcript_path vs agent_transcript_path distinction you flagged. The empty agent_type case is easy to miss too; a matcher that only lists named types would skip exactly the internal ones I wouldn't want to ignore, or worse, trip on them. I'll need to handle that explicitly rather than assuming a type list is enough.
Thanks for laying this out so clearly. This is the kind of hole that would have stayed invisible in my own replay set, because most of those turns weren't delegated.
The drop from 86 blocks down to 2 out of 406 claims explains why most verification controls get turned off. In v0.2.0, roughly 98% of the blocks interrupted valid wrap-ups where a check had already run earlier in the session, which creates enough friction that people uninstall the hook inside a week.
Accepting an earlier-turn check when no code changed afterward trades tidy wrap-up formatting for keeping the guardrail enabled. The remaining gap you pointed out is that an exit code of zero only shows a command finished, leaving open whether it exercised the edited branch. Trying to prove full test coverage from transcript replay hits diminishing returns fast, so blocking the silent no-run claim right after an edit captures most of the downside without bogging down normal turns.
Built a crypto media site and an on-chain analytics tool. Now I invest in crypto and follow post-quantum security. Nights I vibe code and dig Reddit for startup ideas.
Yeah, that remaining gap is the one I'm least happy about. Exit 0 only means a command finished, not that it touched the branch I just edited. I poked at inferring that from the transcript and it fell apart pretty quickly, too many ways a test command can look.
So I stuck with blocking the silent no-run right after an edit. That's the case that actually wasted my time, and it's cheap enough that I leave the hook on. A later check that the command even mentioned the edited files might be worth it, but I'd still leave real coverage to CI.
CTO building Swarmery, an open-source local control plane for Claude Code: one dashboard for all your agent sessions, approvals and plans. Go + React, built in public.
The rule that an honest "not verified yet" always passes is the part I like most. I came to something similar from the other side: in my setup a check that couldn't give an answer, like a timeout or a diff it couldn't read, is a separate state, "don't know", not a pass and not a fail. Before that, "done", "not checked" and "the process just died" all showed the same green badge. About the gap you're least happy with, exit 0 not proving the command touched the edited code: I don't think the transcript can answer that either. What helped me was moving the question one level up. A separate read-only agent checks each phase against the plan, and the tests are written from the plan, not by the agent after its own code. It doesn't prove coverage, but at least someone asks whether the right thing was tested.
Built a crypto media site and an on-chain analytics tool. Now I invest in crypto and follow post-quantum security. Nights I vibe code and dig Reddit for startup ideas.
That distinction between “don’t know” and pass or fail is useful, especially when a timeout or unreadable diff makes the evidence incomplete. My Stop hook is deliberately narrower: it catches a confident “it’s fixed” claim when I can’t find proof in the session, but an exit code still can’t tell me whether the right code was tested. I like the read-only plan check as a separate layer rather than making the hook pretend it can prove coverage.
Staff Software Engineer @ Navan, ML Flow & Cypress Ambassador, Docker Captain and creator of agent-inspect, writing about production AI agents, TypeScript, observability, testing, and developer tools.
The mixed-status escape hatch seems worth a replay fixture: "The redirect is fixed. Deployment isn't verified yet" contains a verified-sounding claim and an honest caveat about a different task. If the caveat clears the whole reply, the first claim never needs evidence. Would scoping "not verified" to the specific claim sentence catch this while keeping your one-block limit and low false-positive rate?
Built a crypto media site and an on-chain analytics tool. Now I invest in crypto and follow post-quantum security. Nights I vibe code and dig Reddit for startup ideas.
That's a good fixture to add. Right now a "not verified" anywhere in the reply can clear the whole thing, even when it's sitting next to a confident claim about something else. I'm leaning toward scoping the caveat to the claim it belongs to. The case I want to be careful with is "the redirect is fixed. I'll verify the deploy next" — the second sentence is a plan, and I don't want that to force a block on a first claim that was already checked. I'll run it through a replay and see where it breaks.
the 77 wrap ups with no tool calls is the bit that stuck with me, the summary is the only thing you actually read so if the proof isnt in it it may as well not exist. same worry with travel answers in searchspot, a confident line with nothing checked behind it is the dangerous one
Built a crypto media site and an on-chain analytics tool. Now I invest in crypto and follow post-quantum security. Nights I vibe code and dig Reddit for startup ideas.
That's the part I didn't expect either. The summary is the only thing I actually read, so if the proof isn't in it, I treat it as unverified. Now I look for the command and its result in the same message before I believe it. Thanks for reading it that closely.
@gapmine agreed, and that is the gap to close. A 'fixed' that arrives with the command and the last lines it printed can be checked; without the tail, '0 tests collected' passes the same way a real run does. Requiring that output tail in the claim closes exactly that hole.
Built a crypto media site and an on-chain analytics tool. Now I invest in crypto and follow post-quantum security. Nights I vibe code and dig Reddit for startup ideas.
Agreed. A clean exit can still hide a run that collected zero tests. I'd want the receipt to include the test count and the last output too, not just the command and exit code. That's much harder to mistake for a real pass.
A 'fixed' is only checkable when it arrives with the command that ran and the last lines it printed. Without those, the claim is the agent grading its own work.
Built a crypto media site and an on-chain analytics tool. Now I invest in crypto and follow post-quantum security. Nights I vibe code and dig Reddit for startup ideas.
That's a clean way to put it, thanks Paul. The part I'm still unsure about is the printed lines. My check only looks at whether a command ran after the last edit, so a run that printed "0 tests collected" looks the same as a real pass. Requiring the output tail in the claim would catch that one.
We're a place where coders share, stay up-to-date and grow their careers.
For the exit-0 gap, the check that has worked best for me costs one extra run: take the edit back out, run the same command, and it has to fail. If it still passes without the fix, it never exercised the fix, whatever the exit code said. Then put the edit back. It doesn't need the transcript at all, and the agent can do it itself before it says "fixed".
The claim that bit me yesterday was "released". The PR was green, the merge was green, and the publish job that runs after the merge had failed on an unrelated flaky test, so the registry still served the previous version. A link to the CI run would have passed your hook, and for that claim it proves the wrong job. What settled it was asking the registry which version it serves. For "deployed", the proof lives at the destination, not in the pipeline.
Thanks, both of these are better than what the hook does today. The "take the edit back out and it has to fail" check is the one I keep wishing I could ask the agent to do by default. My worry is slow test suites, where a second run costs real time, so I'd probably only ask for it when the claim is about a bug fix.
The "released" example is a good catch. You're right that a green CI link proves the wrong job there. The hook only checks that something ran, not that it looked at the destination, so asking the registry which version it serves is a gap I can't close with a transcript check alone.
The set-phrase gap is the one I'd worry about, because an agent can say "done" a hundred ways. In our pipeline at Oizom we gate the action instead of the words: the agent ends its work by calling endWork, and that call is refused until its plan is finished, its stack is healthy and the merge is clean. Your replay over 406 real claims is the part I'd copy. Did the 77 wrap-ups cluster at the end of long sessions, or spread evenly?
I didn't split the 77 by session length, so I can't say yet. I only tagged them as wrap-ups where the check had run in an earlier turn. That's a useful next cut, especially to see whether longer sessions are where the stale "done" signal shows up.
Fair, that cut can wait. From the same replay I'd want to know about the 2 that v0.3.0 still blocks: were they real cases where nothing ran, or false alarms on a phrase? Two out of 406 is small enough to read by hand.
Accepting "a shell command, an MCP tool, a URL fetch, or a subagent" as the thing that ran leaves one path unchecked. Per the hooks docs, Stop runs when the main agent finishes responding, and a subagent finishing fires SubagentStop instead. The plugin's hooks.json registers only Stop. So if the main agent delegates the test run and the subagent reports "all tests pass" without running them, nothing checks that reply, and the Agent call then counts as the successful run for the main turn's claim.
SubagentStop looks like a fit for the same script. Its input carries
last_assistant_messagewith the subagent's final text, it accepts the same block decision as Stop, and the block reason is delivered to the subagent as its next instruction. Two details in that section need handling.transcript_pathis still the main session's transcript there, and the subagent's own file is inagent_transcript_path. The event also fires for internal agents such as prompt suggestions, whereagent_typeis the session's agent name or an empty string, and a matcher that names agent types skips the empty ones.If Stop only runs when the main agent finishes, then a delegated test run would sneak through exactly the way you described. The Agent call itself can look like a successful run, and the subagent's "all tests pass" never gets checked.
I'll look at adding SubagentStop with the same script, and I'll be careful about the transcript_path vs agent_transcript_path distinction you flagged. The empty agent_type case is easy to miss too; a matcher that only lists named types would skip exactly the internal ones I wouldn't want to ignore, or worse, trip on them. I'll need to handle that explicitly rather than assuming a type list is enough.
Thanks for laying this out so clearly. This is the kind of hole that would have stayed invisible in my own replay set, because most of those turns weren't delegated.
The drop from 86 blocks down to 2 out of 406 claims explains why most verification controls get turned off. In v0.2.0, roughly 98% of the blocks interrupted valid wrap-ups where a check had already run earlier in the session, which creates enough friction that people uninstall the hook inside a week.
Accepting an earlier-turn check when no code changed afterward trades tidy wrap-up formatting for keeping the guardrail enabled. The remaining gap you pointed out is that an exit code of zero only shows a command finished, leaving open whether it exercised the edited branch. Trying to prove full test coverage from transcript replay hits diminishing returns fast, so blocking the silent no-run claim right after an edit captures most of the downside without bogging down normal turns.
Yeah, that remaining gap is the one I'm least happy about. Exit 0 only means a command finished, not that it touched the branch I just edited. I poked at inferring that from the transcript and it fell apart pretty quickly, too many ways a test command can look.
So I stuck with blocking the silent no-run right after an edit. That's the case that actually wasted my time, and it's cheap enough that I leave the hook on. A later check that the command even mentioned the edited files might be worth it, but I'd still leave real coverage to CI.
The rule that an honest "not verified yet" always passes is the part I like most. I came to something similar from the other side: in my setup a check that couldn't give an answer, like a timeout or a diff it couldn't read, is a separate state, "don't know", not a pass and not a fail. Before that, "done", "not checked" and "the process just died" all showed the same green badge. About the gap you're least happy with, exit 0 not proving the command touched the edited code: I don't think the transcript can answer that either. What helped me was moving the question one level up. A separate read-only agent checks each phase against the plan, and the tests are written from the plan, not by the agent after its own code. It doesn't prove coverage, but at least someone asks whether the right thing was tested.
That distinction between “don’t know” and pass or fail is useful, especially when a timeout or unreadable diff makes the evidence incomplete. My Stop hook is deliberately narrower: it catches a confident “it’s fixed” claim when I can’t find proof in the session, but an exit code still can’t tell me whether the right code was tested. I like the read-only plan check as a separate layer rather than making the hook pretend it can prove coverage.
The mixed-status escape hatch seems worth a replay fixture: "The redirect is fixed. Deployment isn't verified yet" contains a verified-sounding claim and an honest caveat about a different task. If the caveat clears the whole reply, the first claim never needs evidence. Would scoping "not verified" to the specific claim sentence catch this while keeping your one-block limit and low false-positive rate?
That's a good fixture to add. Right now a "not verified" anywhere in the reply can clear the whole thing, even when it's sitting next to a confident claim about something else. I'm leaning toward scoping the caveat to the claim it belongs to. The case I want to be careful with is "the redirect is fixed. I'll verify the deploy next" — the second sentence is a plan, and I don't want that to force a block on a first claim that was already checked. I'll run it through a replay and see where it breaks.
the 77 wrap ups with no tool calls is the bit that stuck with me, the summary is the only thing you actually read so if the proof isnt in it it may as well not exist. same worry with travel answers in searchspot, a confident line with nothing checked behind it is the dangerous one
That's the part I didn't expect either. The summary is the only thing I actually read, so if the proof isn't in it, I treat it as unverified. Now I look for the command and its result in the same message before I believe it. Thanks for reading it that closely.
@gapmine agreed, and that is the gap to close. A 'fixed' that arrives with the command and the last lines it printed can be checked; without the tail, '0 tests collected' passes the same way a real run does. Requiring that output tail in the claim closes exactly that hole.
Agreed. A clean exit can still hide a run that collected zero tests. I'd want the receipt to include the test count and the last output too, not just the command and exit code. That's much harder to mistake for a real pass.
A 'fixed' is only checkable when it arrives with the command that ran and the last lines it printed. Without those, the claim is the agent grading its own work.
That's a clean way to put it, thanks Paul. The part I'm still unsure about is the printed lines. My check only looks at whether a command ran after the last edit, so a run that printed "0 tests collected" looks the same as a real pass. Requiring the output tail in the claim would catch that one.