DEV Community

Angel
Angel

Posted on Originally published at luisosio.dev

I built a RAG that refuses to answer — and measured when it should

Most RAG demos share a failure mode nobody demos: they always answer. On a
corpus of medical evidence, that stops being a UX quirk and becomes a
dangerous one.

I built a dietary-supplements evidence RAG over 17,698 evidence-grade
PubMed articles. It answers only when the retrieved studies support an
answer, stamps every claim with the PMID it came from, and abstains at the
corpus boundary instead of guessing.

The part worth writing about isn't the pipeline — it's how I know the
refusals are right.

The system, in one glance

gate → hybrid retrieve (BM25 + dense, RRF) → answer contract
     → deterministic verify + judge → one repair pass
     → answer with citations, or abstain
Enter fullscreen mode Exit fullscreen mode

Extractive-first: the default lane quotes study text instead of generating
prose. Every run persists its full trace — gate decision, retrieved chunks,
per-claim stamps, verdicts — at a public URL anyone can inspect.

The eval contract

Two question sets, both committed to the repo.

golden.jsonl — 126 stratified questions. This is what I iterate against.

heldout.jsonl — 199 disjoint questions. This one only measures, never
tunes. When golden and held-out disagree, I've overfit.

The questions come from real sources (SuppLLMent effectiveness items,
MedQuAD consumer questions), with one admission rule: a question enters the
pool only if BM25 actually retrieves evidence for it. 4,026 made it in;
1,543 were rejected — each with the reason kept in rejected.jsonl. A
question nobody can answer in the corpus isn't a test, it's noise.

What the measurement found

On the golden run, every abstention was correct — zero dangerous answers
slipped through. That's the headline.

The less comfortable finding: about 45% of answerable rows abstained too.
Partly the verifier being too strict, partly non-deterministic generation.
So I added a rescue pass — when the system abstains but evidence sits at
the top of the pack, it re-asks whether that evidence actually answers the
question. Advice-decline rows went from 20% to 36% rescued; answerable rows
stayed around 22%, because some of them legitimately lack coverage.

That's the real shape of this work. Less "make the model better," more:
measure a failure class, find the root cause, fix it, re-measure on a set
you didn't tune against.

Try it, check it

The live demo runs on my site — ask it about a supplement and watch it
cite real studies, or watch it refuse. The repo and both eval sets are
public.

I'm an AI engineer in Colombia working remotely (UTC-5). If you're building
systems where "I don't know" is a feature, I'd genuinely like to compare
notes.

Top comments (17)

Collapse
 
gde03 profile image
Giulio D'Erme •

Your 45% is close to mine, but the cause is different: your over-abstains had weak retrieval, and mine had strong retrieval. On 500 LongMemEval questions, a cosine threshold refused 48% of the answerable ones, even though the right session was in the top 5 for 97% of them. Six score signals (cosine, cross-encoder, RRF, QNLI entailment, margin, ratio) all failed to separate those from near-miss unanswerable questions; the best AUC was 0.75.
They measure whether the corpus is about the question, not whether it states the answer. So a step that reads the text, like your extractive lane, is where the decision belongs.

The question set matters too. On one corpus, separability was 0.98 on questions I wrote and 0.72 on real agent queries. Have you compared abstention on live demo traffic against golden?

Collapse
 
agentsearchhq profile image
AgentSearch •

The golden vs held-out split with a "measures, never tunes" rule is the part most eval write-ups skip, so this was good to see.

Curious about the ~45% over-abstain on answerable rows: did you check whether those cluster by question type (dosage vs efficacy, say) or by retrieval rank? If the supporting study sits at rank 1-2 and the verifier still rejects it, that usually means the claim-to-quote match is too literal. A paraphrase-tolerant check on just those rows might rescue more without moving the dangerous-answer rate, which you can confirm on held-out.

Also +1 on publishing the full trace per run. It makes a refusal something you can argue with instead of a mystery.

Collapse
 
losiochico1 profile image
Angel •

Numbers on the verifier-strictness subset: 6 of the 22 over-abstains had decent retrieval (0.50–0.83 best-chunk). Four of those die on a single signal, has_rel near zero (0.04–0.20) while condition and relevance scores are strong, so the strictness class you guessed is real, just narrow. The other 16 average 0.33 retrieval, which is the weak-evidence bulk.

Collapse
 
agentsearchhq profile image
AgentSearch •

Thanks for actually running the numbers. So the bulk is a coverage problem, not a verifier problem, which is the better news since that's fixable on the corpus side. The 4-row has_rel cluster is the interesting bit: strong condition and relevance but near-zero relation presence sounds like the relation check missing hedged phrasing ("associated with", "improved markers of"). Might be worth eyeballing those four quotes by hand.

Collapse
 
losiochico1 profile image
Angel •

Checked this. The split is on retrieval score, not question type. Answerable rows that abstained average 0.33 best-chunk, answered ones average 0.82. Almost all of them are "Is X effective for Y?" questions on niche entities the corpus barely covers. So a paraphrase-tolerant check would only rescue the few high-score cases (0.83 max), not the bulk.

Collapse
 
recallrundev profile image
RecallRun • • Edited

The admission rule (a question only enters if BM25 finds evidence) is a good call, but it also means none of your answerable rows test the case where dense retrieval finds support BM25 missed. If any of the 45% over-abstains are paraphrase-heavy questions, the gate may be stricter than the pool suggests.

Did you log which stage each abstention came from, the gate or the verifier? On a much smaller set I ran (30 answerable and 30 unanswerable questions over the FastAPI docs), a top-1 cosine floor of 0.40 refused 24 of 30 unanswerable and still answered 29 of 30 answerable. The misses left over were near-domain questions whose scores overlapped the lowest answerable ones, so no threshold fixed them. I'd be curious whether your rescued rows look like that overlap or like verifier strictness.

Collapse
 
losiochico1 profile image
Angel •

All 72 refusals are post-gate. Every unclear row passed the gate as evidence intent and died at the evidence check with zero claims extracted; the gate refused nothing on golden. And your blind-spot point is now empirical: in a 50-row no-coverage set I ran, one "unanswerable" question got answered because dense retrieval found what the BM25 admission check missed (it was creatine, which the corpus covers well).

Collapse
 
recallrundev profile image
RecallRun •

That's a cleaner answer than I expected: if the gate refused nothing and all 72 died with zero claims extracted, the abstentions are coming from the claim step, not from retrieval.

One split I'd run on those 72: did the passage that answers the question make it into the retrieved set? If it did and the extractor still pulled zero claims, that's verifier strictness. If it didn't, it's a retrieval miss that just happens to surface at the evidence check. Those two need different fixes.

The creatine row reads to me like a label problem more than a model problem: dense retrieval was right and the "no coverage" tag was wrong. Worth pulling it out of the unanswerable set, or it'll count against a system that answered correctly.

Collapse
 
anne_jacobs_ac166227aa92b profile image
AJ •

The 45% false-abstention finding is the most interesting part to me. Reporting where the system is too cautious makes this much more useful than a “zero hallucinations” headline.

One thing I’d push back on: if questions only enter the eval pool when BM25 retrieves evidence, how are you testing the corpus boundary? I’d want a separate set of genuinely unanswerable questions—including ones with closely related but insufficient evidence—to test whether the system refuses for the right reasons.

That would also help distinguish “the corpus doesn’t cover this” from “retrieval missed it” and “the verifier rejected a supported answer.” Your emphasis on deterministic abstention elsewhere makes that distinction especially interesting. luisosio

Collapse
 
losiochico1 profile image
Angel •

Ran this before replying. Pulled 50 questions the admission rule rejected for no corpus coverage and put them through the pipeline: 48 refused. The 2 that didn't are the interesting part. One was mislabeled, corpus actually covers creatine and the dense leg found what the BM25 check missed. The other is a real failure: phytase-for-infant-development answered on an animal weight-gain study. Chased it to root cause: ~5% of the corpus is animal-only research and PubMed tags animal RCTs with the same "Randomized Controlled Trial" pubtype the ingest filter trusts, so nothing in the pipeline can see species.

Across all eval sets 4 answers cite animal-only studies; 3 present them as human-relevant, 1 hedges correctly (alfalfa dosage explicitly says the studies are farm-animal trials). So your three-way distinction is measured: "not covered" refuses at 96%, "retrieval missed" surfaced as the mislabel, and "related but insufficient" is the species class, logged in KNOWN_GAPS.

Collapse
 
izgorodin profile image
Edward Izgorodin •

This test is the number the headline needs. Take the creatine row out and 49 remain, and one of them, phytase, got an answer from an animal study. One in 49 is about 2%, with a one-sided 95% upper bound of about 9%. The zero on golden comes from questions that passed the BM25 admission rule, a different population, and on the same one-sided scale zero in N only bounds the rate at about 3 in N. Every answer in the no-coverage set is either a failure or a label to fix, as creatine was, so each one needs a hand check, and the rest of rejected.jsonl is the obvious place to grow the set from.

Collapse
 
paullovelace profile image
Paul Lovelace •

zero dangerous answers getting through is a great headline, but the 45% over-abstain is the number I'd be living with — a system that refuses nearly half the questions it could answer is safe, sure, but would anyone keep using it? the rescue pass closing some of that gap is the most interesting part to me

Collapse
 
ascendra_ventures_92c21f2 profile image
Ascendra Ventures •

Keeping a held-out set that only measures and never tunes is the discipline most RAG write-ups skip, and the admission rule (a question enters only if BM25 retrieves evidence for it) is a clever way to keep the boundary honest.

A related lesson from a build at Ascendra Ventures for a federal ministry. The answers officials needed already existed across parliamentary question archives, departmental portals, PDFs and transcripts, but keyword search failed whenever a question was worded differently from the document. The platform combined collectors, document parsing, embeddings and a knowledge graph linking schemes, places and programmes, and the output was a grounded draft with citations rather than a fluent paragraph. The important part was that every answer pointed back to something an official could open and check, so a question that used to take a day of compiling came back as a sourced draft.

One thing I am curious about with your admission rule: how do you count the cases where the corpus did hold the answer but retrieval missed it? Those look like correct abstentions from the outside.

Collapse
 
sara_builds profile image
Sara •

The admission rule for the question pool is something I wish more eval write-ups included — dropping items the corpus can't support, and keeping the rejection reason, makes the abstention numbers actually mean something.

I also liked the extractive-first default. When I've let a generator paraphrase evidence, the citations look fine until I check the quote and the claim has quietly drifted. Starting from the study text seems like a simpler way to keep that honest.

After the rescue pass, the remaining ~22% abstain on answerable rows — do those mostly fail because the retrieved chunk is close but not quite sufficient, or because the verifier still wants a tighter claim-to-sentence match?

Collapse
 
losiochico1 profile image
Angel •

The first branch. n_claims is 0 on every unclear row, meaning the extractive lane found no sentence worth quoting. The scores back it up: abstains average 0.33 retrieval vs 0.82 for answers. Close-but-insufficient, not verify-too-strict.

The exception worth naming: a 6-row subset with good retrieval still abstains on a near-zero relation-presence signal, so verifier strictness does exist, it's just not the bulk.

Collapse
 
enoch_hou profile image
Enoch Hou •

拒绝回答这个设计很有意思,尤其是在医疗证据这种场景下,宁可不答也不能瞎答。45% 的可回答问题也被拒绝这个数字很诚实,好奇救援步骤之后这个比例降到了多少。

Collapse
 
losiochico1 profile image
Angel •

On golden after the rescue, 22 of 50 answerable rows still abstain. The rescue targets advice-decline rows specifically (those went from 20% to 36% rescued). Evidence-insufficiency abstains stay on purpose since some questions genuinely lack coverage.