rag is a local search tool for coding agents. It indexes the files you point it at, searches them by exact words (BM25) and by meaning, and returns the passages that answer a question, with file and line. It runs as a command, rag, and as an MCP server, so Claude Code, Codex and other agents can call it.
uv tool install ragmcp-cli # or: pipx install ragmcp-cli · npm install -g ragmcp-cli
rag connect claude # or codex, gemini, copilot
This post is about measuring it. I ran rag on a public benchmark, found where it lost answers, and changed four things. Correct answers went from 66.3% to 81.0% on the same 499 questions.
The benchmark
EnterpriseRAG-Bench, from Onyx, is a set of questions over a made-up company's documents: Slack threads, email, tickets, docs and code. Each question has a gold answer and the documents that hold it. I used a sample of 100,721 documents and all 499 questions. GPT-6.1 Sol wrote the answers from rag's passages, and the benchmark's own judge prompt graded them.
What lost answers
I read the failures of the first run (rag 0.2) one by one. Two causes covered most of them:
- The right document was found, but not in the top few. Its title words did not count, word forms did not match ("deploy" against "deployments"), and one long document could fill the whole shortlist.
- The right passage was found, but the answer needed the rest of a short document. A ticket's resolution sits three paragraphs below the passage that matched.
What changed
- Step 1: titles are indexed with each passage, words are stemmed, and the shortlist is balanced so one document cannot take every place. Recall in the top 10 rose from 80.2% to 84.6%.
- Step 2: a document of 8,000 characters or less is read whole instead of as a passage.
| Correct | |
|---|---|
| rag 0.2 | 66.3% |
| + titles, stemming, balanced shortlist | 75.2% |
| + short documents read whole (rag 0.3) | 81.0% |
Each step was compared with the one before on the same questions (paired McNemar test). Step 1 fixed 57 questions and broke 13; step 2 fixed 35 and broke 6. Both are well past chance (p < 0.001).
What did not work
Two tries made things worse, and I kept the numbers:
- A softer refusal rule ("answer from partial evidence") cut refusals from 65 to 1, and correct answers fell from 81.0% to 78.6%. Guessing lost more than it gained.
- rag's own answer prompt at first told the model to point out what the evidence did not cover. It scored 79.1%: the answers called settled facts "pending" or added conditions the documents did not state. 0.3.1 drops that wording and scores 80.9%, the same as the benchmark's prompt.
Limits
- The sample is 100,721 of the full 511,962 documents. A larger corpus is harder; the dataset paper's best system scores 68.8% on the full set, so the two numbers are not like for like.
- I designed the changes after reading failures on these same questions. A second benchmark, FreshStack, is next, to check that the gains are not tuned to this one.
No key needed
0.3.1 also changed the first run. Exact-word search works with no account and no key. Without an answer model, a question returns the best passages with file and lines, which is what an agent reads anyway. If you are signed in to Codex with a ChatGPT plan, rag writes answers with it; an OpenRouter key works too.
The code, the experiment record and every number above: https://github.com/MasihMoafi/rag-mcp
Top comments (0)