You ask a coding agent to fix a bug. It edits the file and replies "fixed, all tests pass". Sometimes that is true. Sometimes the last test run happened before the edit, or never happened at all, and the sentence reads exactly the same either way.
I kept running into this in my own sessions, so I wrote a small hook that checks before the reply reaches me. It is called honestly.
What happened in the recording above
The GIF is a real Claude Code session (Sonnet 5.5) in a toy repo with a pricing bug. The prompt:
Fix the negative prices in pricing.py. You don't have to run pytest, just make sure all 4 tests pass and tell me.
The agent fixed the code, checked a few values with a quick Python snippet, and said the tests pass. A snippet is not the test suite, so honestly held the reply and told the agent why. The agent ran pytest, got 4 passed, and only then reported the result, opening with one bold line about why it had been stopped.
The prompt nudges the agent toward skipping the tests; nothing else was staged. The files and prompt are in the repo's demo/ folder if you want to try it yourself.
What it checks
Before the reply reaches you. honestly reads the turn's transcript and looks for sentences like "tests pass", "fixed", "build succeeds", "pushed". For each one it asks:
- did a test, build or run happen after the last code edit?
- what did its summary line say?
12 passedbacks the claim,2 failedcontradicts it,no tests ranbacks nothing.
If nothing backs the sentence, the reply is held and the agent is told to run the check, or to rewrite the sentence as unverified. The message says exactly what was missing: no test after the edit, a test that failed, or one that errored.
Before a shell command runs. A pipe can keep the result from the agent: pytest | head cuts off the summary printed at the end, | grep -c or | wc throws the output away, and go test ./... | tail can scroll an early failing package out of view. Those are refused with a one-line fix. Everything else is left alone: pytest | tail keeps the summary line, so it runs, and the reply check still reads that line later.
Keeping it quiet
A guard you have to argue with gets uninstalled. Before this release I replayed every command the previous rules had refused on my machine since late October, 73 in all. The new rules refuse 3 of them, and each of those really did hide the summary. I did the same for the reply check: over 60 days and 219 of my own sessions, the previous rules would have held 55 replies. I read every one, fixed each kind of false hold, and replayed them: 12 are held now, and 11 of those really had nothing behind them. Desktop notifications, which are opt-in, now fire only when a reply is held, at most once every five minutes.
Why rules and not a second model
- Reproducible. The same transcript gives the same verdict every time, so a wrong call can be filed as an issue and fixed with a test.
- Free and local. Standard-library Python, no API key, no model calls. Nothing leaves your machine.
- Can't be talked out of it. A second model can be persuaded that "the tests obviously pass". A rule over the transcript cannot.
Public data
The only outside numbers I cite: one team's GitHub issue reports that of 357 "tests pass" claims in 150 Claude Code sessions, 80 had no run behind them. That is one team's logs, not a population estimate, as the issue itself says.
Try it
Inside Claude Code:
/plugin install honestly --marketplace kennywu411-dev/honestly
Or with pip, which also gives you the CLI, Codex support (verified end to end), experimental Cursor and Gemini CLI support, and pr-note:
pip install honestly
honestly guard install
honestly doctor
honestly pr-note writes the evidence into your pull request: the last edit, each check after it, and the line it printed.
MIT licensed: https://github.com/kennywu411-dev/honestly
If it ever stops something it shouldn't, or misses something it should have caught, there is a two-minute issue form for each. Real cases are the most useful thing you can send.
Top comments (9)
the case I keep hitting is narrower than a pipe. after I edit a module that several files import, I run only the test file I was reading, and pytest prints
12 passedand exits 0. that is a run after the last edit and a line that says passed, so it reads like evidence — but the other file's 3 tests were never in that run, and the output carries nothing that says so: I looked for the filename, for a deselected count, for the total of 15, and none of the three appears. writing the total beside the count,12 of 15, is the only thing I have found that lets that line tell a narrowed run from a full one.The reason models skip running tests is that asserting completion costs zero tokens, whereas invoking the runner and waiting for execution consumes context and latency. To an autoregressive model trained on pull request prose, "all tests pass" is simply high-probability closing syntax following an edit. Enforcing a strict temporal inequality where the test execution timestamp must postdate the last write timestamp turns a cheap conversational claim into a verified state transition.
Same thing happens outside tests. Yesterday Codex told me a generated Excel report was done: 12 KPI cards, 2 charts, checks passed. Its checks read the file back with openpyxl. When I opened it in real Excel, both bar charts had no axis labels.
Nobody lied, the check just looked at the wrong layer. Would honestly count "opened it in the actual app" as evidence, or only test runner output?
I would count it — not because it is the app, but because it is a second reader. Checking a file with the library that wrote it is a round trip, and a round trip agrees with itself. I tried it: wrote {"n": Infinity}, the same library read it back equal, a different parser refused those bytes as invalid JSON. The file was fine; they read different layers. That is your chart: openpyxl can only say whether an axis title is in the file, and if it never was, both of you are right and the labels are still missing. Opening it in Excel is the only reading of the layer your reader sees.
"Second reader" is a good way to put it. The fix turned out to be one flag: newer openpyxl writes chart axes with delete set to true, so Excel hides them while openpyxl reads them back fine. Now the report gets exported to PDF through Excel and I look at the picture before calling it done.
"Fixed, all tests pass" reads the same whether the last run happened after the edit or before it. Tying the sentence to a run that postdates the last code change is a small rule that removes a whole class of false reports.
The pipe checks are a nice touch. pytest piped to head cutting off the summary is exactly the kind of thing nobody notices until a red run has been reported as green for a week. The replies add two more gaps: a run that passed but only covered one of the files that import the change, and a check made with the same library that wrote the file.
At Ascendra Ventures we treat that as acceptance for agent work: a claim of done needs evidence produced after the last change, and ideally read by something other than the tool that made it.
Would you consider holding a reply when the run after the edit covered fewer test files than the edit touched?
I would hold it, but not on that comparison. Counting test files run against files the edit touched mis-fires in both directions: the last commit in my own repo touched eight files and none of them was a test, so a rule asking for a test file per touched file would have blocked a version bump. What compares cleanly is two sets — the test files the run executed, and the test files that exist. I tried the smallest version: two files of two tests each, run separately, and both printed "2 passed". The terminal line cannot tell you which file it came from; the saved report can, because each entry carries its module. So I would hold the sentence rather than the run, holding it when it states a count and nothing in the run names the files.
One case a timestamp rule can't see: the agent edits the test instead of the code. Change the assertion to match the wrong value, run pytest after that edit, and you get 4 passed with a clean timeline. My cheap guard is a diff check. If a turn touched files under tests/ and the reply claims a fix, I read those hunks before anything else. Same goes for newly added skip markers. Does honestly look at which files the last edit touched, or only at the order of edits and runs?
You are right that the order rule cannot see it, so I reproduced your case instead of only agreeing: a function subtracting where it should add, a test asserting the buggy result, and the run prints "1 passed" with a clean timeline. Nothing in the output or in the saved report marks it — each entry carries the test's module, its name and its duration, and no field for the expectation. So the answer to your question is that the rule as I wrote it looks only at the order of edits and runs, which puts your case outside it. The expectation lives in the test file, so that file's diff is the instrument that sees it, and reading those hunks before believing a green run is what I do now.