Your 45% is close to mine, but the cause is different: your over-abstains had weak retrieval, and mine had strong retrieval. On 500 LongMemEval questions, a cosine threshold refused 48% of the answerable ones, even though the right session was in the top 5 for 97% of them. Six score signals (cosine, cross-encoder, RRF, QNLI entailment, margin, ratio) all failed to separate those from near-miss unanswerable questions; the best AUC was 0.75.
They measure whether the corpus is about the question, not whether it states the answer. So a step that reads the text, like your extractive lane, is where the decision belongs.
The question set matters too. On one corpus, separability was 0.98 on questions I wrote and 0.72 on real agent queries. Have you compared abstention on live demo traffic against golden?
Web search, Extract (page to markdown) and Render for AI agents. Pay per call in USDC via native x402 on Base. No account, no API key. Also on Pocket Network.
The golden vs held-out split with a "measures, never tunes" rule is the part most eval write-ups skip, so this was good to see.
Curious about the ~45% over-abstain on answerable rows: did you check whether those cluster by question type (dosage vs efficacy, say) or by retrieval rank? If the supporting study sits at rank 1-2 and the verifier still rejects it, that usually means the claim-to-quote match is too literal. A paraphrase-tolerant check on just those rows might rescue more without moving the dangerous-answer rate, which you can confirm on held-out.
Also +1 on publishing the full trace per run. It makes a refusal something you can argue with instead of a mystery.
Numbers on the verifier-strictness subset: 6 of the 22 over-abstains had decent retrieval (0.50–0.83 best-chunk). Four of those die on a single signal, has_rel near zero (0.04–0.20) while condition and relevance scores are strong, so the strictness class you guessed is real, just narrow. The other 16 average 0.33 retrieval, which is the weak-evidence bulk.
Web search, Extract (page to markdown) and Render for AI agents. Pay per call in USDC via native x402 on Base. No account, no API key. Also on Pocket Network.
Thanks for actually running the numbers. So the bulk is a coverage problem, not a verifier problem, which is the better news since that's fixable on the corpus side. The 4-row has_rel cluster is the interesting bit: strong condition and relevance but near-zero relation presence sounds like the relation check missing hedged phrasing ("associated with", "improved markers of"). Might be worth eyeballing those four quotes by hand.
Checked this. The split is on retrieval score, not question type. Answerable rows that abstained average 0.33 best-chunk, answered ones average 0.82. Almost all of them are "Is X effective for Y?" questions on niche entities the corpus barely covers. So a paraphrase-tolerant check would only rescue the few high-score cases (0.83 max), not the bulk.
The admission rule (a question only enters if BM25 finds evidence) is a good call, but it also means none of your answerable rows test the case where dense retrieval finds support BM25 missed. If any of the 45% over-abstains are paraphrase-heavy questions, the gate may be stricter than the pool suggests.
Did you log which stage each abstention came from, the gate or the verifier? On a much smaller set I ran (30 answerable and 30 unanswerable questions over the FastAPI docs), a top-1 cosine floor of 0.40 refused 24 of 30 unanswerable and still answered 29 of 30 answerable. The misses left over were near-domain questions whose scores overlapped the lowest answerable ones, so no threshold fixed them. I'd be curious whether your rescued rows look like that overlap or like verifier strictness.
All 72 refusals are post-gate. Every unclear row passed the gate as evidence intent and died at the evidence check with zero claims extracted; the gate refused nothing on golden. And your blind-spot point is now empirical: in a 50-row no-coverage set I ran, one "unanswerable" question got answered because dense retrieval found what the BM25 admission check missed (it was creatine, which the corpus covers well).
That's a cleaner answer than I expected: if the gate refused nothing and all 72 died with zero claims extracted, the abstentions are coming from the claim step, not from retrieval.
One split I'd run on those 72: did the passage that answers the question make it into the retrieved set? If it did and the extractor still pulled zero claims, that's verifier strictness. If it didn't, it's a retrieval miss that just happens to surface at the evidence check. Those two need different fixes.
The creatine row reads to me like a label problem more than a model problem: dense retrieval was right and the "no coverage" tag was wrong. Worth pulling it out of the unanswerable set, or it'll count against a system that answered correctly.
The 45% false-abstention finding is the most interesting part to me. Reporting where the system is too cautious makes this much more useful than a “zero hallucinations” headline.
One thing I’d push back on: if questions only enter the eval pool when BM25 retrieves evidence, how are you testing the corpus boundary? I’d want a separate set of genuinely unanswerable questions—including ones with closely related but insufficient evidence—to test whether the system refuses for the right reasons.
That would also help distinguish “the corpus doesn’t cover this” from “retrieval missed it” and “the verifier rejected a supported answer.” Your emphasis on deterministic abstention elsewhere makes that distinction especially interesting. luisosio
Ran this before replying. Pulled 50 questions the admission rule rejected for no corpus coverage and put them through the pipeline: 48 refused. The 2 that didn't are the interesting part. One was mislabeled, corpus actually covers creatine and the dense leg found what the BM25 check missed. The other is a real failure: phytase-for-infant-development answered on an animal weight-gain study. Chased it to root cause: ~5% of the corpus is animal-only research and PubMed tags animal RCTs with the same "Randomized Controlled Trial" pubtype the ingest filter trusts, so nothing in the pipeline can see species.
Across all eval sets 4 answers cite animal-only studies; 3 present them as human-relevant, 1 hedges correctly (alfalfa dosage explicitly says the studies are farm-animal trials). So your three-way distinction is measured: "not covered" refuses at 96%, "retrieval missed" surfaced as the mislabel, and "related but insufficient" is the species class, logged in KNOWN_GAPS.
Building persistent memory for AI agents at Mnemoverse.
I write about the unglamorous half: forgetting, contradiction, and the moment an agent confidently repeats a fact you retracted.
This test is the number the headline needs. Take the creatine row out and 49 remain, and one of them, phytase, got an answer from an animal study. One in 49 is about 2%, with a one-sided 95% upper bound of about 9%. The zero on golden comes from questions that passed the BM25 admission rule, a different population, and on the same one-sided scale zero in N only bounds the rate at about 3 in N. Every answer in the no-coverage set is either a failure or a label to fix, as creatine was, so each one needs a hand check, and the rest of rejected.jsonl is the obvious place to grow the set from.
I research AI tools, APIs & agents — comparing specs, pricing and real user reports — then publish honest rankings so you don't waste money. Research-first, never sponsored fluff.
zero dangerous answers getting through is a great headline, but the 45% over-abstain is the number I'd be living with — a system that refuses nearly half the questions it could answer is safe, sure, but would anyone keep using it? the rescue pass closing some of that gap is the most interesting part to me
Keeping a held-out set that only measures and never tunes is the discipline most RAG write-ups skip, and the admission rule (a question enters only if BM25 retrieves evidence for it) is a clever way to keep the boundary honest.
A related lesson from a build at Ascendra Ventures for a federal ministry. The answers officials needed already existed across parliamentary question archives, departmental portals, PDFs and transcripts, but keyword search failed whenever a question was worded differently from the document. The platform combined collectors, document parsing, embeddings and a knowledge graph linking schemes, places and programmes, and the output was a grounded draft with citations rather than a fluent paragraph. The important part was that every answer pointed back to something an official could open and check, so a question that used to take a day of compiling came back as a sourced draft.
One thing I am curious about with your admission rule: how do you count the cases where the corpus did hold the answer but retrieval missed it? Those look like correct abstentions from the outside.
The admission rule for the question pool is something I wish more eval write-ups included — dropping items the corpus can't support, and keeping the rejection reason, makes the abstention numbers actually mean something.
I also liked the extractive-first default. When I've let a generator paraphrase evidence, the citations look fine until I check the quote and the claim has quietly drifted. Starting from the study text seems like a simpler way to keep that honest.
After the rescue pass, the remaining ~22% abstain on answerable rows — do those mostly fail because the retrieved chunk is close but not quite sufficient, or because the verifier still wants a tighter claim-to-sentence match?
The first branch. n_claims is 0 on every unclear row, meaning the extractive lane found no sentence worth quoting. The scores back it up: abstains average 0.33 retrieval vs 0.82 for answers. Close-but-insufficient, not verify-too-strict.
The exception worth naming: a 6-row subset with good retrieval still abstains on a near-zero relation-presence signal, so verifier strictness does exist, it's just not the bulk.
On golden after the rescue, 22 of 50 answerable rows still abstain. The rescue targets advice-decline rows specifically (those went from 20% to 36% rescued). Evidence-insufficiency abstains stay on purpose since some questions genuinely lack coverage.
We're a place where coders share, stay up-to-date and grow their careers.
Your 45% is close to mine, but the cause is different: your over-abstains had weak retrieval, and mine had strong retrieval. On 500 LongMemEval questions, a cosine threshold refused 48% of the answerable ones, even though the right session was in the top 5 for 97% of them. Six score signals (cosine, cross-encoder, RRF, QNLI entailment, margin, ratio) all failed to separate those from near-miss unanswerable questions; the best AUC was 0.75.
They measure whether the corpus is about the question, not whether it states the answer. So a step that reads the text, like your extractive lane, is where the decision belongs.
The question set matters too. On one corpus, separability was 0.98 on questions I wrote and 0.72 on real agent queries. Have you compared abstention on live demo traffic against golden?
The golden vs held-out split with a "measures, never tunes" rule is the part most eval write-ups skip, so this was good to see.
Curious about the ~45% over-abstain on answerable rows: did you check whether those cluster by question type (dosage vs efficacy, say) or by retrieval rank? If the supporting study sits at rank 1-2 and the verifier still rejects it, that usually means the claim-to-quote match is too literal. A paraphrase-tolerant check on just those rows might rescue more without moving the dangerous-answer rate, which you can confirm on held-out.
Also +1 on publishing the full trace per run. It makes a refusal something you can argue with instead of a mystery.
Numbers on the verifier-strictness subset: 6 of the 22 over-abstains had decent retrieval (0.50–0.83 best-chunk). Four of those die on a single signal, has_rel near zero (0.04–0.20) while condition and relevance scores are strong, so the strictness class you guessed is real, just narrow. The other 16 average 0.33 retrieval, which is the weak-evidence bulk.
Thanks for actually running the numbers. So the bulk is a coverage problem, not a verifier problem, which is the better news since that's fixable on the corpus side. The 4-row has_rel cluster is the interesting bit: strong condition and relevance but near-zero relation presence sounds like the relation check missing hedged phrasing ("associated with", "improved markers of"). Might be worth eyeballing those four quotes by hand.
Checked this. The split is on retrieval score, not question type. Answerable rows that abstained average 0.33 best-chunk, answered ones average 0.82. Almost all of them are "Is X effective for Y?" questions on niche entities the corpus barely covers. So a paraphrase-tolerant check would only rescue the few high-score cases (0.83 max), not the bulk.
The admission rule (a question only enters if BM25 finds evidence) is a good call, but it also means none of your answerable rows test the case where dense retrieval finds support BM25 missed. If any of the 45% over-abstains are paraphrase-heavy questions, the gate may be stricter than the pool suggests.
Did you log which stage each abstention came from, the gate or the verifier? On a much smaller set I ran (30 answerable and 30 unanswerable questions over the FastAPI docs), a top-1 cosine floor of 0.40 refused 24 of 30 unanswerable and still answered 29 of 30 answerable. The misses left over were near-domain questions whose scores overlapped the lowest answerable ones, so no threshold fixed them. I'd be curious whether your rescued rows look like that overlap or like verifier strictness.
All 72 refusals are post-gate. Every unclear row passed the gate as evidence intent and died at the evidence check with zero claims extracted; the gate refused nothing on golden. And your blind-spot point is now empirical: in a 50-row no-coverage set I ran, one "unanswerable" question got answered because dense retrieval found what the BM25 admission check missed (it was creatine, which the corpus covers well).
That's a cleaner answer than I expected: if the gate refused nothing and all 72 died with zero claims extracted, the abstentions are coming from the claim step, not from retrieval.
One split I'd run on those 72: did the passage that answers the question make it into the retrieved set? If it did and the extractor still pulled zero claims, that's verifier strictness. If it didn't, it's a retrieval miss that just happens to surface at the evidence check. Those two need different fixes.
The creatine row reads to me like a label problem more than a model problem: dense retrieval was right and the "no coverage" tag was wrong. Worth pulling it out of the unanswerable set, or it'll count against a system that answered correctly.
The 45% false-abstention finding is the most interesting part to me. Reporting where the system is too cautious makes this much more useful than a “zero hallucinations” headline.
One thing I’d push back on: if questions only enter the eval pool when BM25 retrieves evidence, how are you testing the corpus boundary? I’d want a separate set of genuinely unanswerable questions—including ones with closely related but insufficient evidence—to test whether the system refuses for the right reasons.
That would also help distinguish “the corpus doesn’t cover this” from “retrieval missed it” and “the verifier rejected a supported answer.” Your emphasis on deterministic abstention elsewhere makes that distinction especially interesting. luisosio
Ran this before replying. Pulled 50 questions the admission rule rejected for no corpus coverage and put them through the pipeline: 48 refused. The 2 that didn't are the interesting part. One was mislabeled, corpus actually covers creatine and the dense leg found what the BM25 check missed. The other is a real failure: phytase-for-infant-development answered on an animal weight-gain study. Chased it to root cause: ~5% of the corpus is animal-only research and PubMed tags animal RCTs with the same "Randomized Controlled Trial" pubtype the ingest filter trusts, so nothing in the pipeline can see species.
Across all eval sets 4 answers cite animal-only studies; 3 present them as human-relevant, 1 hedges correctly (alfalfa dosage explicitly says the studies are farm-animal trials). So your three-way distinction is measured: "not covered" refuses at 96%, "retrieval missed" surfaced as the mislabel, and "related but insufficient" is the species class, logged in KNOWN_GAPS.
This test is the number the headline needs. Take the creatine row out and 49 remain, and one of them, phytase, got an answer from an animal study. One in 49 is about 2%, with a one-sided 95% upper bound of about 9%. The zero on golden comes from questions that passed the BM25 admission rule, a different population, and on the same one-sided scale zero in N only bounds the rate at about 3 in N. Every answer in the no-coverage set is either a failure or a label to fix, as creatine was, so each one needs a hand check, and the rest of rejected.jsonl is the obvious place to grow the set from.
zero dangerous answers getting through is a great headline, but the 45% over-abstain is the number I'd be living with — a system that refuses nearly half the questions it could answer is safe, sure, but would anyone keep using it? the rescue pass closing some of that gap is the most interesting part to me
Keeping a held-out set that only measures and never tunes is the discipline most RAG write-ups skip, and the admission rule (a question enters only if BM25 retrieves evidence for it) is a clever way to keep the boundary honest.
A related lesson from a build at Ascendra Ventures for a federal ministry. The answers officials needed already existed across parliamentary question archives, departmental portals, PDFs and transcripts, but keyword search failed whenever a question was worded differently from the document. The platform combined collectors, document parsing, embeddings and a knowledge graph linking schemes, places and programmes, and the output was a grounded draft with citations rather than a fluent paragraph. The important part was that every answer pointed back to something an official could open and check, so a question that used to take a day of compiling came back as a sourced draft.
One thing I am curious about with your admission rule: how do you count the cases where the corpus did hold the answer but retrieval missed it? Those look like correct abstentions from the outside.
The admission rule for the question pool is something I wish more eval write-ups included — dropping items the corpus can't support, and keeping the rejection reason, makes the abstention numbers actually mean something.
I also liked the extractive-first default. When I've let a generator paraphrase evidence, the citations look fine until I check the quote and the claim has quietly drifted. Starting from the study text seems like a simpler way to keep that honest.
After the rescue pass, the remaining ~22% abstain on answerable rows — do those mostly fail because the retrieved chunk is close but not quite sufficient, or because the verifier still wants a tighter claim-to-sentence match?
The first branch.
n_claimsis 0 on every unclear row, meaning the extractive lane found no sentence worth quoting. The scores back it up: abstains average 0.33 retrieval vs 0.82 for answers. Close-but-insufficient, not verify-too-strict.The exception worth naming: a 6-row subset with good retrieval still abstains on a near-zero relation-presence signal, so verifier strictness does exist, it's just not the bulk.
拒绝回答这个设计很有意思,尤其是在医疗证据这种场景下,宁可不答也不能瞎答。45% 的可回答问题也被拒绝这个数字很诚实,好奇救援步骤之后这个比例降到了多少。
On golden after the rescue, 22 of 50 answerable rows still abstain. The rescue targets advice-decline rows specifically (those went from 20% to 36% rescued). Evidence-insufficiency abstains stay on purpose since some questions genuinely lack coverage.