Gabby Gill

Posted on Nov 11 Edited on Nov 11

Build Buddy, Day 5: my vector search's best match was a wrong answer

#ai #postgres #rag #buildinpublic

Day 5 of MLH Global Hack Week: Builders Week. All week, Build Buddy has been collecting explanations from AI models about C# compiler errors. Some of them are right. Some are confidently wrong. Today I put them in a vector database to see what happens when you search them like a RAG system would.

The setup

12 real explanations of compiler errors from Gemma 3 4B and Grok 4.7, collected during the week
Each one reviewed by hand: correct, wrong, or inconsistent
Embedded locally with nomic-embed-text through Ollama, so no text left my laptop
Stored in a Tiger Data hypertable with a VECTOR(768) column and an HNSW index

CREATE TABLE fix_notes (
    noted_at    TIMESTAMPTZ NOT NULL,
    model       TEXT        NOT NULL,
    error_code  TEXT        NOT NULL,
    explanation TEXT        NOT NULL,
    review      TEXT        NOT NULL,
    embedding   VECTOR(768) NOT NULL
);
SELECT create_hypertable('fix_notes', by_range('noted_at', INTERVAL '1 day'));
CREATE INDEX ON fix_notes USING hnsw (embedding vector_cosine_ops);

Then I asked questions in plain English and ranked every explanation by cosine distance (lower means closer in meaning).

When it works, it works well

"An async call returns a Task and is assigned to a string without await."

The top match was Grok's explanation: "GetStringAsync returns Task<string>, so the call is awaited." Distance 0.245. The next best was 0.338.

It even told apart two errors with the same code. Gemma had also explained a CS0029, but that one was about assigning a string to an int. It ranked #7. Matching on the error code alone would have treated them as the same.

The best match was wrong

"A statement is missing its semicolon at the end of the line."

#	Model	Code	Review	Distance
1	gemma3:4b	CS1002	wrong	0.231
2	grok-4.7	CS0161	correct	0.318
3	gemma3:4b	CS0161	correct	0.331

The closest match, by a clear margin, was Gemma's explanation that the semicolon "wasn't required." That's the opposite of the real error.

Vector search found exactly the right topic. It has no idea whether the content is true. A RAG system would have retrieved this and repeated it with confidence.

Two filters, two different jobs

Filter 1: trusted sources only. Adding WHERE review = 'correct' removed the wrong answer. But there was no correct semicolon explanation in my data, so the search returned the closest leftovers instead: notes about missing return values, at 0.318 and up. Still an answer, just an irrelevant one.

Filter 2: a distance threshold. Every real match in my data scored between 0.186 and 0.261. Everything unrelated scored 0.318 or more. So a threshold around 0.30 catches "nothing relevant found."

But the threshold alone isn't enough. The wrong semicolon answer scored 0.231, comfortably inside it.

So retrieval needs both:

Trusted sources, so wrong answers can't be retrieved
A relevance threshold, so unrelated answers aren't used
And when nothing passes both: "I don't have a trusted answer."

Time filtering worked the same way. WHERE noted_at >= '2026-11-09' removed Grok's Nov 8 async explanation, and the remaining results were all weak.

Limits

12 explanations, 5 questions, one embedding model, and reviews done by me. The 0.30 threshold fits this data only. It's a demonstration, not a benchmark.

Full rankings, data, and scripts: db/fix-notes-results.md

Update: the behaviour checks are in (issue #1). A live rerun gave 6/6 compiled, 5/6 behaved.

Today's lesson shows up there too. The CS1002 and CS1061 fixes now behave correctly, but their explanations are still wrong. The code passes; the words don't. Checking explanations is a possible next step.

How it was built: the schema, embedding script, and analysis were worked through with Claude; I ran everything on my laptop and in the Tiger Data console, and every number comes from those runs.

Top comments (2)

KolmoPDF • Nov 11

Adding a metadata filter like WHERE review = 'correct' solves the trust gap, though it also exposes a subtle HNSW quirk in Postgres once the table grows. Because standard pgvector HNSW indexes apply WHERE clauses as post-filters after traversing the graph, a query can exhaust its ef_search candidate budget on rows marked 'wrong' and return fewer results than expected. Creating a partial HNSW index (CREATE INDEX ... WHERE review = 'correct') or enabling hnsw.iterative_scan keeps only verified notes in the candidate graph.

Cosine distance behaving identically on opposite claims ("missing semicolon" vs "semicolon wasn't required" landing at 0.231) shows up constantly in technical document ingestion too. Embedding models cluster around shared domain vocabulary and mostly flatten negation or conditional constraints, so separating retrieval similarity from verification status at index time is usually the cleanest boundary.

Gabby Gill • Nov 12

Great points, thank you. I checked the pgvector docs and you're right: HNSW applies the WHERE as a post-filter, so with the default ef_search of 40, a selective filter like review = 'correct' can return fewer rows than the LIMIT once the table grows.

With only 12 rows it couldn't bite here: the candidate list is bigger than the table, and I ran the full rankings as exact scans to be sure. But a partial index fits this case well, since review only has three values:

CREATE INDEX ON fix_notes USING hnsw (embedding vector_cosine_ops) WHERE review = 'correct';

and iterative scans (pgvector 0.8+) as a fallback for other filters.

The negation point is exactly what stuck with me too. "Missing semicolon" and "semicolon wasn't required" share almost every word, so the embedding puts them together. Keeping "is it relevant" and "is it verified" as separate checks seems like the only safe boundary.

DEV Community