Last week I wrote about 7 ways AI code is "almost right". At the end I asked you for an 8th way and promised to share the best answers here, with credit.
I asked for one. You gave me seven: five brand new ones, and two that make checks from my list sharper. Mudassir also shared a simple rule from his team, it's near the end. Honestly, a few of your replies taught me more than my own post did. Thank you to everyone who took the time to write them out, some of these clearly came from a bad day in production.
Quick confession before we start. The cover for that post came from an image generator, and on the first try it dropped one word from my headline. The picture looked great. The sentence was wrong. I caught it just before saving. Kind of embarrassing, considering what the post was about.
8. Right shape, wrong value
Maxwell's comment is the one that worries me most. No exception, nothing to catch, the field is filled in. It's just not what the source said. In his words, "a parser can read annual as monthly across a whole layout and never once fail."
30-second check (his): pick 5 rows at random and compare them with the source by hand. Random matters. If you only check the rows that already looked odd, you catch nothing.
I'd add two things: check the labels and units in those rows too, and if the source has a total you can trust (a column sum, a row count), compare it. A matching total doesn't prove every field is right, but a wrong conversion usually shows up there first.
9. You asked for one line. It tidied three files.
Aliza Ali calls this silent scope creep. You ask for a one-line fix, and the diff also renames a variable, "cleans up" an import and touches files you never mentioned. On their own, none of those changes looks worth worrying about. Together they change a lot more than you asked for.
Her follow-up made a point I hadn't thought about: auto-accept makes this much more likely, but step-by-step review doesn't fully save you either, because each edit looks reasonable in isolation. The final diff is the only view that shows the whole footprint at once.
30-second check: run git diff --stat before reading a single line (or git diff --stat main...HEAD if the agent already committed). More files isn't automatically bad, since a real fix can need its test and a type too. But ask about every file you didn't expect before reviewing anything else.
10. State that drifts
This one is harder to spot, because the code reads clean. Khali Sollis described a custom hook or async routine that passes the first render test but quietly misses a reactive dependency, or closes over stale state. It "works fine during a quick manual check, but drifts out of sync the second real users start interacting with the UI sequentially."
30-second check: don't test it once after a fresh load. Change the state a few times in a row, and trigger anything that runs later (timers, async callbacks). Also look for a newly added eslint-disable for react-hooks/exhaustive-deps in the diff. That rule catches many of these, and silencing it is the quickest way to make the warning go away.
11. The test that would have caught it was mocked away
mp3sss sees this in Electron work. An AI-written IPC handler looks correct but never validates what the renderer sends, and the integration test that should catch that is mocked. Same story around auto-update and crash reporting: the happy path is easy to generate, and the failure path (network down, partial download, a stale crash dump) is exactly what gets skipped.
Mudassir Khan hit the same wall from another side: a catch that returns an empty array passes every test, because the same agent wrote the tests.
30-second check: for each AI change, force one failure on purpose. Cut the network, send a bad payload, break the dependency, and make sure you get an error instead of a quiet empty result. As mp3sss put it, "pairing each AI change with a forced failure-path test has saved us more than the happy-path tests did."
12. The flaky test that got a longer timeout
Max Kaya added this one to the test checks from #2. Besides toBe quietly turning into toBeTruthy, watch for a timeout extended "to fix flakiness". In his words, it "usually means the test now passes by construction."
Sometimes the old timeout really was too tight for CI. Often, though, the slow call or the race is still there, and you've just stopped hearing about it.
30-second check: search the diff for bigger timeout numbers and new retries (retries, jest.retryTimes, --retries). Each one should point to a timing problem someone actually understands.
13. A real package, but far too much of it
Artur Pokładek called this a cousin of #3. The package exists and is popular, it just pulls in a pile of dependencies for something that was fifteen lines of code. He accepted one from an agent without looking.
30-second check (his): git diff --stat on the lockfile. If it grew by hundreds of lines for a small feature, ask why. A big lockfile diff can have boring causes too, so treat it as a reason to look, not as proof.
I asked him what he does then, and his answer is a better rule than anything I had. A small helper, he writes himself and tests. Dates, parsing and crypto stay with a well-known library, because there his own fifteen lines would be the bug. The small lookalike package is the option he trusts least.
14. It passes everything, and never ships
Guillermo Baigorria had this happen the day before he commented. The AI fixed a bug with a new patch folder, and every test passed on his machine. But the deploy script packs a fixed list of folders, and the new one wasn't on it. No test could see that, because tests run where the file exists.
Paul Lovelace called it the gap between where tests run and what actually ships, and said this is the check he'd steal first.
The list is hard-coded on purpose, by the way. A fixed list is what keeps a stray .env from shipping. So Guillermo's team checks the built package instead of the script, flags new files, and a person decides.
30-second check (his): list the files the change added, then look for each one in what you ship.
git diff --name-status main | grep '^A'
Compare that with your Dockerfile COPY lines or the files list in your package.json.
A rule worth copying
Mudassir also sent a defence against #2 from last time. His point: when the test file changed in the same diff, a green run tells you very little. It shows the new test passes, not that the old behaviour is still checked. So his team's rule is that any diff touching a test file needs a one-line reason in the PR, and their review bot fails a test that got skipped or loosened to toBeDefined.
What I like is how cheap it is. Whoever changed the test just says why. And if nobody can explain why a test changed, that's a good reason to stop and look closer.
Your turn again
If I had to pick the worst, it's #8. A fix that never shipped usually gets noticed once someone hits the bug again. A wrong number in a report can quietly survive for months.
Which of these have you actually run into? And is anyone catching any of them automatically in CI, without someone reading every diff? If the comments grow like last time, maybe there's a part three in here.
Small note: Mudassir's rule reminded me of my free Claude Code plugin, proof-of-done. It flags test changes that make checks pass without a fix (skipped tests, weaker assertions, deleted tests, a new eslint-disable or @ts-ignore) and won't let the agent say "done" until its report explains them. Since the last post it also checks each package in a monorepo on its own, which came from an earlier comment by Mudassir, and points out changed lines your tests never ran. Watch the 30-second demo.
Everything I've built (all free and open source)
which-ai-to-use: picks the best AI and model for each task, and hands your work to another AI when you hit a usage limit
- Try it:
npx which-ai-to-use - npm · VS Code Marketplace · Open VSX (Cursor, Windsurf) · GitHub · Website
proof-of-done: a Claude Code plugin that won't let the AI say "done" until the tests pass
- Install:
/plugin marketplace add WST2002/claude-code-skillsthen/plugin install proof-of-done@claude-code-skills - GitHub · Featured on the HOL blog
security-audit: a Claude Code skill that finds security holes before attackers do
All my Claude Code skills: github.com/WST2002/claude-code-skills
Watch the demos on YouTube: @wst2002rk · proof-of-done Short · which-ai-to-use Short · security-audit Short
Read more: 7 ways AI code is "almost right" · Switch AIs mid-task without losing context · Your AI said "fixed" but never ran the tests · My AI finds security holes before attackers do
Find me: GitHub · YouTube · LinkedIn · Portfolio · Hashnode · Peerlist · npm · LeetCode · Email
Top comments (12)
An eighth from my side, earned the slow way: the code handles exactly the case you described, and quietly decides the case you left unstated. No error, no TODO, just a reasonable-looking choice buried in the middle — a default, a sort order, an empty-input behaviour — that you only discover weeks later when it disagrees with what you actually wanted. It's the hardest almost-right to catch because review checks what you asked for. The unstated decisions are the ones nobody reviews, including the person who wrote the prompt. My fix is dull: before a run, I write down my assumptions where the agent can contradict them, and after, I diff the result against that list.
Thank you, this is a sneaky one, because the code is right for everything you actually said. One cheap way to surface those hidden choices is to ask the agent at the end to list every decision it made that the prompt didn't specify: defaults, ordering, what happens on empty input. It won't catch everything, but it turns a buried choice into a line you can disagree with. Do you keep your assumptions in the prompt itself, or in a separate file the agent reads?
Separate file, mostly. The standing assumptions — the ones that are true for every run — live in a rules file the agent reads at the start, so I don't retype them into every prompt and they don't drift between runs. The assumptions specific to one task go in that task's brief, in a short list the agent is told to contradict if they're wrong. And your end-of-run trick is the third piece: the decisions the agent made that nobody specified get written into the run record. So three layers: standing file, per-task list, and the run's own log catching what both missed. The prompt itself is the worst place for assumptions, in my experience — it's the one text nobody re-reads.
Thank you, that three-layer setup makes a lot of sense. A rule in a file gets looked at again every time the file changes, while the same rule buried in a prompt is gone after one run. And using the run log to catch what both lists missed closes the loop nicely.
Glad it landed. One small addition from the scar tissue: the run-log layer only works if it's read on a schedule, not when something already looks wrong. I do a one-minute pass over the logs on weekday mornings and save the real judgement for the weekend. The logs that get read late are just archaeology. The ones read daily actually change the next run.
"Logs read late are just archaeology" is going on a sticky note. I'm taking the one-minute weekday pass too. Do you read the raw logs, or does the agent write you a short summary at the end of each run?
The “right shape, wrong value” one stuck with me most — a filled-in field with no exception is exactly the kind of failure that slips past a green test run. Maxwell’s check of comparing five random rows with the source by hand, labels and units included, is wonderfully cheap. Have you found any of these seven checks lend themselves to running automatically in CI, or do they mostly stay manual 30-second habits?
Thank you! Most of them can move into CI, and the one that can't is the interesting part. Artur's heavy package is the easiest: a bundle size limit fails the build when a new dependency adds too much. Guillermo's new folder that never ships becomes a check of the built output against a list of files that must be there, like Mateo suggested above. Khali's stale closure is mostly caught by the exhaustive-deps lint rule. A raised timeout can be a tiny diff check that flags any timeout value that went up.
Maxwell's right shape, wrong value is the one I'd keep human. A test can compare against a saved sample, but once someone picks the sample rows they stop being random. So the five-row check stays a habit and CI does the rest. Which one would you automate first?
I'd start with Artur's heavy package too — a bundle size limit is the most mechanical of the lot, it fails the build with a clear number attached, and it catches the problem at the exact moment someone adds the dependency. The raised-timeout diff check would be my second pick for the same reason: it's a tiny, objective signal. The five-row check staying human makes sense to me; the moment the sample rows are chosen by hand, the randomness that makes the check work is gone.
Glad we landed on the same two. One small thing I learned with size limits: set the limit just above today's size, not with a big margin. Then it fails on the one commit that adds the heavy package, instead of quietly letting three medium ones through first.
The “passes everything, and never ships” example is particularly important because it exposes a gap that code-level validation often misses. A change can be logically correct, have passing tests, and still be absent from the actual deployment artifact.
At IT Path Solutions, this is one of the areas we pay attention to when stabilizing AI-generated applications. I'd add one more CI check: validate the final build or container against the expected artifact manifest, including newly added files and required configuration. Also, test changes deserve independent scrutiny otherwise, an agent can unintentionally make the implementation and its verification agree on the same wrong assumption. The real definition of done should be that the intended behavior survives the entire path to production.
Thank you, Mateo. Checking the final artifact against a manifest is a nice way to make Guillermo's check automatic: once the expected file list lives in the repo, a missing folder fails CI instead of waiting for someone to read the deploy script. The one thing I'd watch is keeping that manifest up to date, or it turns into one more list nobody remembers to edit. Do you generate it from the build itself, or maintain it by hand?