I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
exact name lookups dropping from 99% to 63% under naive vector reranking is the number that should stop most add vector search proposals. we saw the same shape on a docs search, and keeping the keyword winner on top when its lead was clear fixed identifiers without losing the paraphrase gains. our mistake was tuning fusion weights on our own test queries, so offline numbers beat real traffic. how did you pick the 5% lead threshold, tuned on the benchmark or checked on held out queries?
The decision was made to choose 1.05× from our Wikipedia corpus by asking when BM25's top hit makes sense. An exact name is always implied by a clear lead. Although the results are nearly identical, it is not a knife-edge: anything from 1.0 to 1.1 gives nearly identical results in this case.
We then examined three hold-out sets using human-written queries and judgements. According to SciFact and NFCorpus, fusion with the pin is superior to plain fusion. On HotpotQA, plain fusion without the pin performed best (nDCG@10 0.830 vs 0.822). Since HotpotQA's distractors were selected based on word overlap with the question, this is not surprising. This is exactly what makes a keyword winner untrustworthy. So the pin is a net win for our kind of content, not universal.
Real-traffic is our gap as well: all of this is offline, based on queries that are curated or generated. Logging a sample of real questions from users' wikis and adding them to the held-out set is the next step I'd like to take before tuning anything further.
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
fair, that 0.830 vs 0.822 on HotpotQA is the useful part, since a pin that loses on one set and wins on two is honest evidence instead of a clean story. on the real traffic gap, we sampled about 200 live queries a week and hand labeled only the ones where keyword and vector disagreed on the top hit. that kept labeling small and surfaced the identifier misses fast. are you planning to log just the disagreement cases, or a random sample of everything?
Disagreement cases first. That is a good tip, and it describes how the pin works: only the top hit changes the result. There is no action taken by the pin when BM25 and vector search agree on the top hit.
I would add a small random sample to the top, however. There are instances in which disagreements cannot capture the cases where both methods agree on the same wrong page, and these are the cases in which offline numbers are most effective in hiding.
In contrast to your setup, Synthadoc runs locally on each user's own wiki, so we never see their traffic. The logging of disagreement cases must be opt-in and remain on the user's computer as a local file. Instead of us labeling the pages, the labels would be derived from what users already do in the app, such as which cited pages they open. The held-out set could then be grown by sharing anonymised (question, top hits) pairs. None of this is built yet. Your 200 disagreements per week are a useful benchmark for determining the size of the labeling effort.
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
the random sample of agreed hits is the right call imo. when retrievers agree on a wrong page nothing flags it, so those are the ones that sit in a dashboard looking healthy for months.
for the local only logging, a cited page the user opens and then immediately backs out of is a decent free negative label, no server traffic needed. would a plain exported file the user can choose to share be enough to grow the held out set, or is that too much friction?
That is an excellent idea. A quick bounce does not cost anything to record, but on its own, it is a noisy signal. Often, in Synthadoc, people open a cited page just to check one sentence and then return to it. Using a quick bounce followed by the user rephrasing the same question is a much more reliable "wrong page" indicator, and both events occur in the same session, so the message stays local.
I believe it is sufficient, as long as the export takes only one command and displays exactly what is contained in it: the questions, the top hits by page name, the labels, and no page content. Despite the friction, a few dozen real questions collected from different wikis would still tell us more than another generated set. Rather than replacing the public corpora, it would add to them.
Thank you for sharing your thoughts. The disagreement-plus-random-sample approach is good to be on the roadmap.
Over 30 years of experience in distributed systems, advanced cloud applications, and serverless platforms. Currently focusing on agentic AI, autonomous systems, and multi-agent systems.
Peter, this is a really thorough piece of work. By running retrieval-only (without LLM in the loop), the results are truly reproducible, and catching the stopword bug and the overlapping random seeds before trusting the numbers demonstrates the right approach.
In the keyword-pin strategy, you let BM25 remain authoritative when it is confident, rather than blending everything into a mushy RRF score. I think that is a much cleaner model.
As I read, I kept thinking about one question: in the case of CJK, you're routing to BM25-only because the embedding model is English-only, and DuRetrieval confirmed that it harms rather than helps. Have you tried a multilingual embedding model (e.g., multilingual-e5-small or paraphrase-multilingual-mpnet-base-v2) to see whether hybrid search becomes viable for CJK, or whether BM25 stays competitive even with a proper cross-lingual model? For a mixed-language wiki, I'd be curious if semantic matching ever outweighs latency costs. I think the team could try these things later in the project.
Thanks for pointing out the retrieval-only setup and the keyword pin. We have not yet tried a multilingual model; it is a logical next step. On DuRetrieval, it brought semantic search back to exactly BM25's numbers, with no regression. A multilingual model is the only way to turn that tie into a win.
The good news is that the experiment is cheap now. DuRetrieval is already in the benchmark, so trying a model like multilingual-e5-small or a multilingual MiniLM is mostly a configuration change plus one embedded run. The questions we would want it to answer are:
Does hybrid beat BM25 on Chinese, or does BM25 stay competitive?
Does it fix the case we currently can't handle: an English question about a page written only in Chinese?
What does it cost? These models are several times larger than the 65 MB English one, and query embedding is just most of our added latency (about 20 ms today).
Benchmarking before turning embeddings on by default is the part I would point other teams to. The re-rank result is the warning: paraphrase hits went up while exact name lookups fell from 99% to 63%, which would have made search worse for most people while the feature request looked like an obvious win.
The clear winner rule is a sensible way to let BM25 stay in charge when it is confident. Edward Izgorodin's catch on the threshold mismatch shows why the measurement and the shipped rule need to use the same setting, or the numbers describe a system nobody runs.
Paul Irolla's question about pages sent per answer is the one a budget owner would ask. If semantic results let the agent read fewer pages for the same recall, that saving is worth measuring next to answer quality.
The 18x speedup from caching the index is a good reminder that a careful benchmark often pays for itself through the problems it finds along the way.
Edward's catch turned out to matter. Measured at the 1.05× margin we actually ship, BM25's second and third pinned results were almost never the right page. So the next release pins only the top hit: paraphrase recall@5 goes from 68% to 70%, held-out corpora improve slightly, and exact names stay at 98%. We'll add an update to the post when it ships.
On pages per answer, the benchmark already hints at the saving: semantic search's top 5 finds the right page about as often as keyword search's top 20 (0.90 vs 0.89 recall). Whether the agent can read a quarter of the pages without hurting answers is the next thing to measure, with answer quality alongside retrieval, as you say.
The 1.05x clear-winner rule is a neat fix, and I like that you showed why simply weighting BM25 higher collapsed fusion back into keyword search before landing on it.
One query shape I'd want in the benchmark before trusting the rule on real wikis is the mixed one, where someone uses an exact name but asks a paraphrased question about it, like "what did we end up deciding about Postgres for the billing thing". In a notes wiki that name shows up on lots of pages, so BM25 can have a clear winner that is just the page mentioning Postgres most often, the rule pins it into the top 3, and the page that actually holds the decision never gets its vote from the embedding side. When I logged real questions against a notes wiki, that shape was more common than pure paraphrases or pure lookups.
The other thing I noticed is that generated paraphrases tend to be cleaner than what people type. Real questions are shorter, half remembered, and often borrow a word from a neighbouring page rather than using no page words at all, which is harder for BM25 than "no words from the title". Mixing a small set of questions collected from actual use into the held out set moved my rankings more than tuning thresholds did.
Did you check how often the clear-winner rule fires on paraphrase questions that happen to contain a name, and whether it helps or hurts there?
Thanks, this is exactly the case I'd worry about too, and the answer is: not segmented that way. We measured how often the clear-winner signal fires on paraphrases as a whole (at a 1.2× margin it fired on 12% of them, and BM25's top hit was right 84% of the time), but we never split out paraphrases that contain a name. The cost you describe does show up in the aggregate, though: the pin dropped paraphrase recall@5 from 0.694 (plain fusion) to 0.676, while recovering exact-name lookups from 0.936 to 0.984. Your Postgres example is a plausible explanation for some of those 1.8 points.
Two things soften it in practice. The pin only fixes BM25's top 3, so the embedding side's best page can still land on 4 or 5. And the query agent hands over up to 20 pages to the LLM, where the paraphrase recall@20 is 0.875. So the decision page usually still reaches the model; it's ranked lower than it should be.
The point you raise about generated paraphrases being too clean is fair, and we have included it in our caveats. As human-written checks, we used HotpotQA and two BEIR sets, but neither had your "name + half-remembered question" shape. Adding a mixed query type and a small set of real logged questions would be a good next step for the benchmark. Thanks for the push.
I write when an idea won't leave me alone 🧠 Building AI agents and the tools to build AI agents. Love connecting AI with other fields and yapping about all of them.
The benchmark called search directly, so it never saw the query agent searching only the English translation of each CJK question. I run my retrieval evals that same direct way because it's fast and gives the same numbers every run. Are you adding a few questions that go through the full query path, or does the hand-built mixed wiki stay as that check?
This is the main lesson we learned from that bug. The direct-call benchmark is still the right tool for comparing ranking setups, for exactly the reasons you give: it's fast and deterministic. The query agent, however, is not able to see anything that the query agent does before searching.
Today the end-to-end check is a unit test rather than a benchmark. A mixed Chinese/English wiki is used for the agent's real search step, with real BM25, and only the LLM calls (translation and question splitting) are stubbed with fixed answers. This keeps the behavior deterministic, and it fails on the old translation-only behavior. As a result of the hand-built wiki, the bug was discovered manually.
What I'd like next is a benchmark mode that runs queries through that same agent path, replaying recorded translations and decompositions. That way the numbers stay reproducible but include everything the agent does to a question. Haven't built it yet. How do you handle LLM-side query rewriting in your evals?
Building persistent memory for AI agents at Mnemoverse.
I write about the unglamorous half: forgetting, contradiction, and the moment an agent confidently repeats a fact you retracted.
The table in section 4 that makes the case for the keyword pin is measured at a different threshold from the one the rule uses. It reports how often BM25's top result is right when its best score is at least 1.2 times the second, while the rule keeps BM25's top three on top from 1.05 times. Between 1.05 and 1.2 the pin fires on more questions than that table describes, and the second and third pinned pages have no precision figure at all. The cost shows in your own results: paraphrase Recall@5 is 0.694 with plain fusion and 0.676 with the pin, so on paraphrases, the class the whole change was built for, the pin gives back a little under two points. Reporting how often the 1.05 rule fires per question type, and how often the second and third pinned pages are the labelled one, would show whether pinning only the top hit keeps the identifier gain at a smaller paraphrase cost.
The second place worth a number is gap detection. Whether the wiki covers a question is still judged on keyword evidence alone, which stops a merely similar page from hiding a real gap, but paraphrased questions are exactly the ones with little keyword evidence. A question that fusion now answers from the right page can still be flagged as a gap. Counting, on the paraphrase set, how often the labelled page is in the top 5 while the gap check fires would put a figure on that trade, and paraphrases are the class the original issue was about.
They are both fair, and I have now measured both on the same set of 1,985 pages.
The pin at 1.05× fires far more than the 1.2× table suggests: on 96% of own-word questions, 98% of exact-name ones, and 53% of paraphrases. Those paraphrases from BM25's top hit are right only 45% of the time. Pinned ranks 2 and 3 are almost never on the labelled page (≤ 6%), so pinning them buys little. Your proposal checks out: pinning only the top hit gives paraphrase recall@5 0.699 (vs 0.676 shipped, 0.694 plain fusion) and 0.901 overall (vs 0.896). The catch is the exact names: 0.978 vs 0.984, which takes them 1.6 points under BM25 and past the bar we set. So it's a trade, and a better one than I'd assumed. Next I want to try pinning the top hit at 1.05× and the top three only at a higher margin.
Gap detection: When semantic search finds the right paraphrase page in the top 5 (292 of 432), the gap check still fires on 26 of them, 8.9% (keyword search: 11 of 207, 5.3%). One caveat: the benchmark runs the check on the raw question, while the product runs it on the LLM's rewritten sub-questions, so treat it as indicative. Still, it's a real cost for exactly the class this was built for. Thanks for pushing on both.
Independent builder and OPC operator. Currently building SoloVault — a platform that helps solo founders discover what to build by turning weak market signals into validated startup ideas.
The HotpotQA result being flipped from the other two is a good example of why you need diverse test data. One dataset's assumptions can poison the whole picture.
Thank you, Richard. That is exactly what occurred. In HotpotQA, distractor paragraphs are selected because they share words with the question, so the dataset is designed to make keyword searches difficult. It can be used with any semantic method. The re-rank on HotpotQA increased from 0.717 to 0.842 Recall@5, but on our own wiki-style content, it was not able to recover a single page that BM25 had missed (paraphrase Recall@20 remained at 0.646). We would have shipped the weaker design if HotpotQA had been our only benchmark.
Because of this, the benchmark is built using our own type of content: Wikipedia pages with paraphrased and exact-name questions, at a variety of wiki sizes. Public corpora (HotpotQA, SciFact, NFCorpus, DuRetrieval) serve as cross-checks rather than as deciding factors. Also, the cross-checks were useful: DuRetrieval was the one that demonstrated that English-only embedding made Chinese search worse, which is why CJK queries now only use keyword search.
For a wiki, the bill is usually the pages sent per answer, not the search itself. Did the vector run cut the pages sent per question, or just the misses?
Fair points. Misses are what we measure, not costs. It is the same number of candidate pages sent either way (up to 20, then trimmed to a context budget), so semantic search only changes which pages reach the LLM, not how many.
However, your question points to a potential savings we have not yet realized. Semantic search's top 5 (recall 0.896) matches keyword search's top 20 (0.892) on the 1,985-page benchmark. This implies the same coverage with a quarter of the pages per answer. In terms of pool size, we have not changed it, as it is a matter of balancing quality and cost. The end-to-end quality of the answer would need to be measured, not just the retrieval of the answer. It is a good next experiment, and probably the biggest win for most users than the actual misses.
That is exactly right, and that is why it was not shipped. Reranking sorts BM25's candidates purely by meaning, so a page that matches "Moore School 1946" exactly can drop below pages that are solely concerned with early computers. That is the 99% x 63% you quoted.
Instead, What shipped runs both searches and merges their rankings, keeping BM25's top hit on top whenever there is a clear winner (best score ≥ 1.05× the second). The most popular keyword is almost always the exact name, so exact-name lookups remain at 98%, while paraphrases remain at 48% to 68%. With the keyword search, people who are searching for an exact table or page name are able to find what they are looking for.
@peter-white-1492 Fair, that answers it: the same number of pages go out, only which ones change. Our 0.896 vs 0.892 on the 1,985-page benchmark is retrieval recall, though, not answer quality, so the quarter-of-pages saving stays a hypothesis until the answers are measured.
We're a place where coders share, stay up-to-date and grow their careers.
exact name lookups dropping from 99% to 63% under naive vector reranking is the number that should stop most add vector search proposals. we saw the same shape on a docs search, and keeping the keyword winner on top when its lead was clear fixed identifiers without losing the paraphrase gains. our mistake was tuning fusion weights on our own test queries, so offline numbers beat real traffic. how did you pick the 5% lead threshold, tuned on the benchmark or checked on held out queries?
The decision was made to choose 1.05× from our Wikipedia corpus by asking when BM25's top hit makes sense. An exact name is always implied by a clear lead. Although the results are nearly identical, it is not a knife-edge: anything from 1.0 to 1.1 gives nearly identical results in this case.
We then examined three hold-out sets using human-written queries and judgements. According to SciFact and NFCorpus, fusion with the pin is superior to plain fusion. On HotpotQA, plain fusion without the pin performed best (nDCG@10 0.830 vs 0.822). Since HotpotQA's distractors were selected based on word overlap with the question, this is not surprising. This is exactly what makes a keyword winner untrustworthy. So the pin is a net win for our kind of content, not universal.
Real-traffic is our gap as well: all of this is offline, based on queries that are curated or generated. Logging a sample of real questions from users' wikis and adding them to the held-out set is the next step I'd like to take before tuning anything further.
fair, that 0.830 vs 0.822 on HotpotQA is the useful part, since a pin that loses on one set and wins on two is honest evidence instead of a clean story. on the real traffic gap, we sampled about 200 live queries a week and hand labeled only the ones where keyword and vector disagreed on the top hit. that kept labeling small and surfaced the identifier misses fast. are you planning to log just the disagreement cases, or a random sample of everything?
Disagreement cases first. That is a good tip, and it describes how the pin works: only the top hit changes the result. There is no action taken by the pin when BM25 and vector search agree on the top hit.
I would add a small random sample to the top, however. There are instances in which disagreements cannot capture the cases where both methods agree on the same wrong page, and these are the cases in which offline numbers are most effective in hiding.
In contrast to your setup, Synthadoc runs locally on each user's own wiki, so we never see their traffic. The logging of disagreement cases must be opt-in and remain on the user's computer as a local file. Instead of us labeling the pages, the labels would be derived from what users already do in the app, such as which cited pages they open. The held-out set could then be grown by sharing anonymised (question, top hits) pairs. None of this is built yet. Your 200 disagreements per week are a useful benchmark for determining the size of the labeling effort.
the random sample of agreed hits is the right call imo. when retrievers agree on a wrong page nothing flags it, so those are the ones that sit in a dashboard looking healthy for months.
for the local only logging, a cited page the user opens and then immediately backs out of is a decent free negative label, no server traffic needed. would a plain exported file the user can choose to share be enough to grow the held out set, or is that too much friction?
That is an excellent idea. A quick bounce does not cost anything to record, but on its own, it is a noisy signal. Often, in Synthadoc, people open a cited page just to check one sentence and then return to it. Using a quick bounce followed by the user rephrasing the same question is a much more reliable "wrong page" indicator, and both events occur in the same session, so the message stays local.
I believe it is sufficient, as long as the export takes only one command and displays exactly what is contained in it: the questions, the top hits by page name, the labels, and no page content. Despite the friction, a few dozen real questions collected from different wikis would still tell us more than another generated set. Rather than replacing the public corpora, it would add to them.
Thank you for sharing your thoughts. The disagreement-plus-random-sample approach is good to be on the roadmap.
Peter, this is a really thorough piece of work. By running retrieval-only (without LLM in the loop), the results are truly reproducible, and catching the stopword bug and the overlapping random seeds before trusting the numbers demonstrates the right approach.
In the keyword-pin strategy, you let BM25 remain authoritative when it is confident, rather than blending everything into a mushy RRF score. I think that is a much cleaner model.
As I read, I kept thinking about one question: in the case of CJK, you're routing to BM25-only because the embedding model is English-only, and DuRetrieval confirmed that it harms rather than helps. Have you tried a multilingual embedding model (e.g., multilingual-e5-small or paraphrase-multilingual-mpnet-base-v2) to see whether hybrid search becomes viable for CJK, or whether BM25 stays competitive even with a proper cross-lingual model? For a mixed-language wiki, I'd be curious if semantic matching ever outweighs latency costs. I think the team could try these things later in the project.
Thanks for pointing out the retrieval-only setup and the keyword pin. We have not yet tried a multilingual model; it is a logical next step. On DuRetrieval, it brought semantic search back to exactly BM25's numbers, with no regression. A multilingual model is the only way to turn that tie into a win.
The good news is that the experiment is cheap now. DuRetrieval is already in the benchmark, so trying a model like multilingual-e5-small or a multilingual MiniLM is mostly a configuration change plus one embedded run. The questions we would want it to answer are:
Benchmarking before turning embeddings on by default is the part I would point other teams to. The re-rank result is the warning: paraphrase hits went up while exact name lookups fell from 99% to 63%, which would have made search worse for most people while the feature request looked like an obvious win.
The clear winner rule is a sensible way to let BM25 stay in charge when it is confident. Edward Izgorodin's catch on the threshold mismatch shows why the measurement and the shipped rule need to use the same setting, or the numbers describe a system nobody runs.
Paul Irolla's question about pages sent per answer is the one a budget owner would ask. If semantic results let the agent read fewer pages for the same recall, that saving is worth measuring next to answer quality.
The 18x speedup from caching the index is a good reminder that a careful benchmark often pays for itself through the problems it finds along the way.
Thanks, and agreed on all four points.
Edward's catch turned out to matter. Measured at the 1.05× margin we actually ship, BM25's second and third pinned results were almost never the right page. So the next release pins only the top hit: paraphrase recall@5 goes from 68% to 70%, held-out corpora improve slightly, and exact names stay at 98%. We'll add an update to the post when it ships.
On pages per answer, the benchmark already hints at the saving: semantic search's top 5 finds the right page about as often as keyword search's top 20 (0.90 vs 0.89 recall). Whether the agent can read a quarter of the pages without hurting answers is the next thing to measure, with answer quality alongside retrieval, as you say.
The 1.05x clear-winner rule is a neat fix, and I like that you showed why simply weighting BM25 higher collapsed fusion back into keyword search before landing on it.
One query shape I'd want in the benchmark before trusting the rule on real wikis is the mixed one, where someone uses an exact name but asks a paraphrased question about it, like "what did we end up deciding about Postgres for the billing thing". In a notes wiki that name shows up on lots of pages, so BM25 can have a clear winner that is just the page mentioning Postgres most often, the rule pins it into the top 3, and the page that actually holds the decision never gets its vote from the embedding side. When I logged real questions against a notes wiki, that shape was more common than pure paraphrases or pure lookups.
The other thing I noticed is that generated paraphrases tend to be cleaner than what people type. Real questions are shorter, half remembered, and often borrow a word from a neighbouring page rather than using no page words at all, which is harder for BM25 than "no words from the title". Mixing a small set of questions collected from actual use into the held out set moved my rankings more than tuning thresholds did.
Did you check how often the clear-winner rule fires on paraphrase questions that happen to contain a name, and whether it helps or hurts there?
Thanks, this is exactly the case I'd worry about too, and the answer is: not segmented that way. We measured how often the clear-winner signal fires on paraphrases as a whole (at a 1.2× margin it fired on 12% of them, and BM25's top hit was right 84% of the time), but we never split out paraphrases that contain a name. The cost you describe does show up in the aggregate, though: the pin dropped paraphrase recall@5 from 0.694 (plain fusion) to 0.676, while recovering exact-name lookups from 0.936 to 0.984. Your Postgres example is a plausible explanation for some of those 1.8 points.
Two things soften it in practice. The pin only fixes BM25's top 3, so the embedding side's best page can still land on 4 or 5. And the query agent hands over up to 20 pages to the LLM, where the paraphrase recall@20 is 0.875. So the decision page usually still reaches the model; it's ranked lower than it should be.
The point you raise about generated paraphrases being too clean is fair, and we have included it in our caveats. As human-written checks, we used HotpotQA and two BEIR sets, but neither had your "name + half-remembered question" shape. Adding a mixed query type and a small set of real logged questions would be a good next step for the benchmark. Thanks for the push.
The benchmark called search directly, so it never saw the query agent searching only the English translation of each CJK question. I run my retrieval evals that same direct way because it's fast and gives the same numbers every run. Are you adding a few questions that go through the full query path, or does the hand-built mixed wiki stay as that check?
This is the main lesson we learned from that bug. The direct-call benchmark is still the right tool for comparing ranking setups, for exactly the reasons you give: it's fast and deterministic. The query agent, however, is not able to see anything that the query agent does before searching.
Today the end-to-end check is a unit test rather than a benchmark. A mixed Chinese/English wiki is used for the agent's real search step, with real BM25, and only the LLM calls (translation and question splitting) are stubbed with fixed answers. This keeps the behavior deterministic, and it fails on the old translation-only behavior. As a result of the hand-built wiki, the bug was discovered manually.
What I'd like next is a benchmark mode that runs queries through that same agent path, replaying recorded translations and decompositions. That way the numbers stay reproducible but include everything the agent does to a question. Haven't built it yet. How do you handle LLM-side query rewriting in your evals?
The table in section 4 that makes the case for the keyword pin is measured at a different threshold from the one the rule uses. It reports how often BM25's top result is right when its best score is at least 1.2 times the second, while the rule keeps BM25's top three on top from 1.05 times. Between 1.05 and 1.2 the pin fires on more questions than that table describes, and the second and third pinned pages have no precision figure at all. The cost shows in your own results: paraphrase Recall@5 is 0.694 with plain fusion and 0.676 with the pin, so on paraphrases, the class the whole change was built for, the pin gives back a little under two points. Reporting how often the 1.05 rule fires per question type, and how often the second and third pinned pages are the labelled one, would show whether pinning only the top hit keeps the identifier gain at a smaller paraphrase cost.
The second place worth a number is gap detection. Whether the wiki covers a question is still judged on keyword evidence alone, which stops a merely similar page from hiding a real gap, but paraphrased questions are exactly the ones with little keyword evidence. A question that fusion now answers from the right page can still be flagged as a gap. Counting, on the paraphrase set, how often the labelled page is in the top 5 while the gap check fires would put a figure on that trade, and paraphrases are the class the original issue was about.
They are both fair, and I have now measured both on the same set of 1,985 pages.
The pin at 1.05× fires far more than the 1.2× table suggests: on 96% of own-word questions, 98% of exact-name ones, and 53% of paraphrases. Those paraphrases from BM25's top hit are right only 45% of the time. Pinned ranks 2 and 3 are almost never on the labelled page (≤ 6%), so pinning them buys little. Your proposal checks out: pinning only the top hit gives paraphrase recall@5 0.699 (vs 0.676 shipped, 0.694 plain fusion) and 0.901 overall (vs 0.896). The catch is the exact names: 0.978 vs 0.984, which takes them 1.6 points under BM25 and past the bar we set. So it's a trade, and a better one than I'd assumed. Next I want to try pinning the top hit at 1.05× and the top three only at a higher margin.
Gap detection: When semantic search finds the right paraphrase page in the top 5 (292 of 432), the gap check still fires on 26 of them, 8.9% (keyword search: 11 of 207, 5.3%). One caveat: the benchmark runs the check on the raw question, while the product runs it on the LLM's rewritten sub-questions, so treat it as indicative. Still, it's a real cost for exactly the class this was built for. Thanks for pushing on both.
The HotpotQA result being flipped from the other two is a good example of why you need diverse test data. One dataset's assumptions can poison the whole picture.
Thank you, Richard. That is exactly what occurred. In HotpotQA, distractor paragraphs are selected because they share words with the question, so the dataset is designed to make keyword searches difficult. It can be used with any semantic method. The re-rank on HotpotQA increased from 0.717 to 0.842 Recall@5, but on our own wiki-style content, it was not able to recover a single page that BM25 had missed (paraphrase Recall@20 remained at 0.646). We would have shipped the weaker design if HotpotQA had been our only benchmark.
Because of this, the benchmark is built using our own type of content: Wikipedia pages with paraphrased and exact-name questions, at a variety of wiki sizes. Public corpora (HotpotQA, SciFact, NFCorpus, DuRetrieval) serve as cross-checks rather than as deciding factors. Also, the cross-checks were useful: DuRetrieval was the one that demonstrated that English-only embedding made Chinese search worse, which is why CJK queries now only use keyword search.
For a wiki, the bill is usually the pages sent per answer, not the search itself. Did the vector run cut the pages sent per question, or just the misses?
Fair points. Misses are what we measure, not costs. It is the same number of candidate pages sent either way (up to 20, then trimmed to a context budget), so semantic search only changes which pages reach the LLM, not how many.
However, your question points to a potential savings we have not yet realized. Semantic search's top 5 (recall 0.896) matches keyword search's top 20 (0.892) on the 1,985-page benchmark. This implies the same coverage with a quarter of the pages per answer. In terms of pool size, we have not changed it, as it is a matter of balancing quality and cost. The end-to-end quality of the answer would need to be measured, not just the retrieval of the answer. It is a good next experiment, and probably the biggest win for most users than the actual misses.
That is exactly right, and that is why it was not shipped. Reranking sorts BM25's candidates purely by meaning, so a page that matches "Moore School 1946" exactly can drop below pages that are solely concerned with early computers. That is the 99% x 63% you quoted.
Instead, What shipped runs both searches and merges their rankings, keeping BM25's top hit on top whenever there is a clear winner (best score ≥ 1.05× the second). The most popular keyword is almost always the exact name, so exact-name lookups remain at 98%, while paraphrases remain at 48% to 68%. With the keyword search, people who are searching for an exact table or page name are able to find what they are looking for.
@peter-white-1492 Fair, that answers it: the same number of pages go out, only which ones change. Our 0.896 vs 0.892 on the 1,985-page benchmark is retrieval recall, though, not answer quality, so the quarter-of-pages saving stays a hypothesis until the answers are measured.