The setting that is not a setting
Human-in-the-loop appears in most AI proposals as a checkbox. It is on, the client is reassured, and the conversation moves to the timeline.
Then the system ships, and the review queue turns out to hold four hundred items a day, and by week three someone is clearing it in batches at 6pm without reading. The approval rate is ninety-nine per cent. The audit trail says a human approved every one.
That is worse than having no review, because it manufactures evidence of oversight that did not happen.
The line moves per process
The only question that produces a useful boundary is what a wrong output costs here.
- Reversible, internal, cheap to correct: run unattended and monitor.
- Client-facing, irreversible, or expensive to unwind: a person decides before it takes effect.
Those two categories rarely map onto departments, systems or seniority, which is why a company-wide policy always ends up in the wrong place. A draft internal note and an outbound message to a client are not the same kind of error and cannot take the same rule.
The four behaviors this resolves to in practice are described in the safe AI framework.
What goes in the specification
Written before the build, not after.
What the reviewer sees. Not the raw output. The output plus what is needed to judge it: the source, a confidence signal, how similar cases went last week.
What they are deciding. A question with a small number of answers, not an invitation to have a look.
What happens on rejection. Back to the queue, escalate to a named person, or fall to the manual path. This is the part most often left undefined, and it is the one that decides whether the review is real - a rejection with nowhere to go becomes an approval eventually.
Measuring whether it works
Two numbers, both uncomfortable.
Rejection rate. At or near zero, the review is not happening, and the dashboard cannot tell you the difference between an accurate system and an unread queue.
Median decision time. Four seconds is not a decision.
When both look wrong, the usual cause is not indiscipline. It is that the boundary was drawn too wide and the reviewer is drowning. Narrowing it to the outputs that actually carry cost restores attention to the ones that matter.
What breaks without the stage
Two failure modes, opposite in shape, identical in outcome.
Too tight, and you have built a queue that nobody clears honestly. Too loose, and the first irreversible mistake teaches you where the line should have been, in front of a client.
Both come from the same root: the line was drawn after the system existed, by whoever was in the room when it went wrong. Drawn during Cast, against the measured cost of a wrong output, it is a design decision. Drawn afterwards, it is an apology with a workflow attached.
Where it sits
Cast is the fifth of six stages, and the other half of it is building against a frozen specification rather than against a conversation. What both halves are protecting against is the same thing that sinks most conversational deployments, described in why your last chatbot failed. The whole sequence, applied to one deployment, is in the protocol end to end.
Top comments (0)