Friday AI Question. This week's question is one I think most teams never ask out loud:
When your AI feature doesn't know the answer, what does it actually do?
Pick one (and tell me in the comments):
- A) It always answers. Something is better than nothing.
- B) It refuses only on hard-coded rules (blocked topics, missing permissions).
- C) It abstains when the evidence is weak: says "I couldn't find this", shows what it searched, and offers a next step.
- D) C, plus our evals score "I don't know" separately from wrong answers.
My lean: C on day one, D as soon as you have an eval set. Here's why.
The scoreboard problem
OpenAI's September 2025 post Why language models hallucinate argues that models hallucinate partly because standard training and evaluation reward guessing over acknowledging uncertainty.
Their analogy is a multiple-choice test: if you don't know, a wild guess might be right, while leaving it blank guarantees zero. When models are graded only on accuracy, guessing wins.
They split answers into three buckets: accurate, errors, and abstentions. And they show it with SimpleQA numbers from the GPT-5 system card:
| Metric | gpt-5-thinking-mini | OpenAI o4-mini |
|---|---|---|
| Abstention rate | 52% | 1% |
| Accuracy rate | 22% | 24% |
| Error rate | 26% | 75% |
o4-mini looks slightly better on accuracy (24% vs 22%). But its error rate, its rate of confidently wrong answers, is 75% vs 26%.
If your only metric is "% correct", you'd ship the model that is wrong three times as often.
Why this matters for your feature, not just for leaderboards
Your internal evals are a scoreboard too. If your RAG or support bot eval only asks "was the answer right?", every "I don't know" counts as a failure, and every prompt tweak that makes the model guess more will look like an improvement.
The post's suggested fix: penalize confident errors more than uncertainty, and give partial credit for appropriate expressions of uncertainty. OpenAI's Model Spec says the same thing in product terms: it's better to indicate uncertainty or ask for clarification than to give confident information that may be incorrect.
What option C looks like in practice
- Give the model permission to abstain. Say explicitly in the instructions that "I couldn't find this in the provided documents" is a valid answer.
- Abstain on evidence, not vibes. If retrieval returns nothing relevant, don't hand the model an empty context and hope.
- Make abstention useful. Show what was searched, suggest a rephrase, or hand off to a human.
- Log abstentions. A cluster of "I don't know" on the same topic is a content gap you can fix.
What option D adds
Split your eval results into three numbers instead of one: correct, wrong, abstained. Then decide what a wrong answer costs in your product compared to an abstention. For a medical or billing bot, a wrong answer is much worse than "I don't know". For a brainstorming tool, maybe not.
Once you track errors separately, you can see whether a change made the model smarter, or just bolder.
Your turn
A, B, C or D? And if you track abstentions: what abstention rate do you consider healthy?
References
- OpenAI, Why language models hallucinate (Sep 5, 2025): https://openai.com/index/why-language-models-hallucinate/
- OpenAI Model Spec: https://model-spec.openai.com/
Top comments (3)
D, mostly. The thing I'd add to C is that "evidence is weak" needs a number picked from labelled data, not a feeling.
I tried a top-1 cosine threshold on the FastAPI docs (MiniLM, 2,808 chunks) with 30 answerable and 30 unanswerable questions. At 0.40 it refused 24 of the 30 unanswerable ones and still answered 29 of 30 answerable.
The unrelated questions (boiling eggs, tides) were easy: all of them scored below 0.30, under the weakest real question at 0.39. The hard part was near-domain stuff like rate limiting or Prometheus metrics. Those scored up to 0.51 and overlapped the lowest answerable queries, so no single threshold separates them. That's where your answer-level check has to take over.
One trap: "retrieval returned nothing" doesn't work as the abstain signal with BM25. It returned candidates for all 60 queries, because words like "how" and "do" always match something.
That overlap in the near-domain questions is the useful finding. When no single threshold separates them, I'd route the 0.35-0.55 band to a second check, like asking the model whether the retrieved chunk actually answers the question, and refuse only if that fails too.
C, with one amendment from hard experience: the abstention has to show its work or it decays into laziness with good PR. 'I couldn't find this' plus what was searched, where, and what would settle it — that's a useful answer. A bare 'I don't know' is just a refusal wearing the costume. On scoring: D matters more than it looks. In my world the expensive failure was never silence; it was a confident number that turned out to be invented, steering real money before anyone rechecked it. An eval that lumps abstentions with wrong answers trains exactly that failure — it teaches the model that guessing pays better than honesty. Score 'didn't know and said so' as its own class, and price a confident wrong answer at ten times an abstention. The behaviour you want shows up fast once the scoreboard stops punishing it.