DEV Community

Cover image for Write the End of the Pilot Before It Starts
weiche chiu
weiche chiu

Posted on Originally published at williamlab.dev AI-assisted

Write the End of the Pilot Before It Starts

In my last piece I listed four places where an agent pilot stops. The fourth had the most public numbers behind it and the weakest of them, and in my experience the most consequence: nobody decided what happens next. This piece is about that one. It is a practice piece. I have no new measurements, only a way to write the end of a pilot down before it begins, and a reason to think it matters.

Two doors

In October I wrote four stop conditions for a first pilot: data out of scope, unsupported affirmative answers, bypassed approval, quality below the bar. Those are safety stops. Any of them means halt and investigate. They say nothing about what happens when none of them fires and the pilot simply keeps running.

A pilot that runs without incident has two ways to end on purpose. It graduates and becomes someone's production process. Or it is retired, because the organization decides it is not worth continuing. Both are decisions. In the pilots I have seen, neither one is written down, so neither one gets made. The pilot stays a pilot until a budget round ends it for reasons nobody can trace afterwards.

Gartner's June 2025 prediction that over 40% of agentic AI projects will be canceled by the end of 2027 attributes the cancellations to "escalating costs, unclear business value or inadequate risk controls." It is a forecast, not a count, and last time I argued that each reason is decidable before the pilot starts. This piece is the form that decision takes. Two sentences, each with a date and a signature.

The graduation sentence

"Becomes a process" is vague enough never to happen. Written as a sentence someone can sign, it has four parts and a date.

An owner, by name, who runs the process after the pilot team leaves. Not the sponsor who approved the pilot. The person whose team will file the tickets when it breaks.

A budget line. That needs the number from the second stop in the last piece: what an accepted task cost last month, tokens and human review included. If nobody has it, the sentence cannot be written, and that is information in itself.

A way to switch it off. Who can turn it off, and what the step looks like in the week after. If the answer is that the work goes back to being done by hand, the sentence should say by whom, and whether they still know how.

An acceptance check that runs without the pilot team. The first stop in the last piece was that nobody could accept a result except by reading it. Graduation means the check belongs to the process, not to the people who built the agent.

Take the synthetic support case from the first-pilot piece, a reply draft grounded in a return policy. I have added a support lead and a ticketing system to it; they are invented too. The sentence might read: "On 15 January the support lead takes ownership of the reply-draft step, the policy-version check runs inside the ticketing system, at least N drafts a week passed review in December at a cost per accepted draft at or below Y against the support team's budget line, the support lead can switch the step off by returning drafts to manual, and the pilot team is released." The date, N and Y are placeholders. Every clause is checkable on the day, and that is the point of writing it.

The retirement sentence

The second sentence is harder to write, because it admits the pilot might not be worth it. It needs the same date and the same numbers. "On 15 January, if fewer than N drafts a week passed review in December, or the cost per accepted draft was above Y, the pilot stops and the step returns to manual, with the support lead owning it." Same placeholders. The two conditions should be exact complements, so that on the day one of them is true. If they are not, the sentences should say who decides in the gap.

Pick N and Y before the pilot starts. After the start, in my experience, every number starts to look like a judgment on the people who ran the pilot, and the sentence gets softened until it can no longer fire. Before the start it is only a plan.

One thing I cannot show you with data: whether pilots carry a retirement condition. No survey I know of measures it. The nearest proxy is in McKinsey's 2026 State of AI survey: high performers, the 6% of respondents who attribute at least 5% of EBIT to AI and report "significant" value from it, are twice as likely as others to report that their organizations have defined processes to measure the impact of AI initiatives. That is a measurement process, not a retirement condition, and it is self-reported. In a commentary box on the same page, one contributor describes the ongoing challenge as avoiding "simply accumulating tools or pilots" and building a repeatable capability instead. That is commentary, not a finding. I cite both because they point the same way as what I have seen, not because they prove it.

Who signs, and when

Both sentences are signed by the process owner, the person named in the graduation sentence. Not the AI team, who in my experience have every reason to keep the pilot alive. Not the sponsor, who by the decision date tends to be busy elsewhere. If the owner will not sign the retirement sentence, the pilot has no owner, and you have learned that before spending anything.

Then one date on the calendar, the same date in both sentences, when the owner, the sponsor and whoever ran the pilot read the two sentences and pick one. Not a review meeting with slides. A reading.

What I have seen when this is skipped is not failure. The fourth stop is quiet. The pilot runs, the sponsor reports it as a win, the owner never takes it, and months later a budget round asks what it costs and nobody has the number. The cancellation note then names all three of Gartner's reasons at once, and none of the three was ever decided.

Four fields for the one-page table

The first-pilot piece had a one-page table per workflow, with a row for when to stop. That row covers the safety stops. Add four rows beside it for the end.

Field What to record
Graduation condition The sentence: owner by name, budget line, switch-off path, acceptance check, date
Retirement condition The sentence: the same date, the numbers that stop it, where the work goes afterwards
Decision owner The process owner who signs both sentences
Decision date The one date both sentences name

Write "unknown" in any cell you cannot fill. An unknown in the budget line is the second stop from the last piece. An unknown in the acceptance check is the first. An unknown in the owner is the fourth, and in the pilots I have seen it is the one that ends them.

If your pilot has an end condition, I would like to read it, and to know whether it fired. If it does not, the two sentences take an afternoon. The hard part is not writing them. It is finding who signs.

Top comments (0)