A game search result has a deadline: the player should see a useful ranking before the next click. A private knowledge manager has the same constraint, even when its documents are patch notes, build guides, and personal play logs. Retrieval quality and latency are coupled; optimizing one in isolation produces a system that feels broken.
Short answer: use a bounded local retrieval pass first, reserve expensive reranking for ambiguous top candidates, and make every deadline visible in telemetry.
The before-and-after request path
Picture the request as a short relay race. The query parser normalizes text, a lexical index finds likely game documents, a local vector index widens the candidate set, and a reranker orders only that small set before generation receives evidence. Each runner gets a handoff time. No stage gets to borrow the whole race clock.
Before this split, every query waits for the same heavyweight model. Exact titles pay for semantic work they do not need, while broad queries create a queue behind one another. After the split, exact-name searches can finish in the first tier and only uncertain rankings spend more compute. The important change is predictability, not a magic model.
Deadlines are product behavior.
Start with a policy budget. For a 180 ms interactive target, one reasonable allocation is 15 ms parsing, 55 ms candidate retrieval, 80 ms reranking, and 30 ms evidence assembly. Those are planning numbers, not performance claims. Measure your own device, corpus, and cache state.
I once watched a dashboard report a healthy p50 while a cold index pushed a few requests past 900 ms. The missing field was queue time. Recording queue, retrieval, and rerank timers separately made the decision obvious: when less than 25 ms remained, a second-stage call could only make the response late. That incident also exposed a subtle feedback loop: a slow request held a worker, the next request waited in the same queue, and the resulting timeout looked like a model-quality problem because its candidate list was empty. We fixed the diagnosis by emitting a span when a request entered the queue, then making the fallback decision from the remaining budget rather than from the original 180 ms target. The ranking did not get smarter. The service got honest about what it could finish.
How should privacy, personal knowledge retrieval, and latency budgets shape the architecture?
Keep note text, embeddings, and query traces on the device by default. A remote component, if one is allowed at all, should receive a redacted identifier after a policy check. A trace containing the full query can become an accidental second copy of a private journal, so retention and access controls belong in the retrieval design rather than in a later audit.
Use a narrow, versioned contract between stages. The retriever returns IDs, scores, and a reason code; the reranker returns an ordered list, confidence, and elapsed time. Add a schemaVersion and a maximum candidate count. Those two fields prevent a client update from changing score meaning or turning a bounded lookup into a broad scan.
The gaming corpus makes this concrete. “Starfall 2 patch notes” should use an exact-name path. “co-op survival crafting after the winter update” needs lexical plus vector recall, then perhaps a cross-encoder over the top 20. Both requests follow the same privacy policy, but they do not deserve the same compute.
Small contracts are easier to observe.
type Candidate = {
id: string;
lexicalScore: number;
vectorScore: number;
};
type RerankDecision = {
ordered: Candidate[];
confidence: number;
elapsedMs: number;
tier: "exact" | "vector" | "deep" | "fallback";
};
export function selectTier(
candidates: Candidate[],
remainingMs: number,
): RerankDecision["tier"] {
if (candidates.length === 0) return "fallback";
const first = candidates[0];
const second = candidates[1];
const margin = second ? first.vectorScore - second.vectorScore : 1;
if (remainingMs < 25) return "fallback";
if (margin > 0.18) return "exact";
if (remainingMs < 80) return "vector";
return "deep";
}
The margin is a routing signal, not a calibrated probability. Calibrate it against labeled queries from the game catalog and the personal notebook separately; short game names and private prose have different typo patterns. I'm not sure one threshold will survive a multilingual corpus, so keep the threshold in configuration and record its version.
Three tiers for quality without tail-latency surprises
The first tier is lexical and cached. It handles exact names, aliases, and recently opened notes with a strict candidate cap. The second adds local vector similarity for paraphrases and spelling variation. The third reranks the top 20 with a heavier model when the score margin is narrow and enough budget remains. A final fallback returns the best available evidence with an explicit low-confidence state.
Do not collapse those paths into one latency metric. Emit queue_ms, retrieval_ms, rerank_ms, candidate_count, selected_tier, and policy. A timeout is an observable outcome, not an exception to discard. Trace context can be propagated with the W3C Trace Context format, while OpenTelemetry gives a vendor-neutral shape for logs and metrics.
type RetrievalEvent = {
traceId: string;
selectedTier: "exact" | "vector" | "deep" | "fallback";
queueMs: number;
retrievalMs: number;
rerankMs: number;
candidateCount: number;
policy: "on-device" | "redacted";
status: "ok" | "timeout";
};
export function shouldEscalate(event: RetrievalEvent, budgetMs: number): boolean {
const spent = event.queueMs + event.retrievalMs + event.rerankMs;
return event.status === "ok" && spent < budgetMs && event.candidateCount > 1;
}
The useful alerts are relationships. Rising p95 with a stable candidate count points toward scheduling or storage contention. Falling recall with stable latency points toward index freshness or tokenization. Log a request hash and document IDs, never the note body. Sampling should be deliberate: keep all budget breaches, but sample routine successful traces.
What should you test before widening a retrieval tier?
Build a small golden set with exact game names, misspellings, genre terms, and cross-note privacy boundaries. Measure recall@k and nDCG beside p50, p95, and timeout rate. Retrieval-Augmented Generation research frames retrieval as part of answer quality, so inspect the evidence list before judging generated prose.
Replay the set with a warm cache, a cold cache, and a busy device. Inject a full queue and a stale index. Test an empty result too. The correct behavior is a fast, explicit “no matching evidence,” not a confident answer assembled from unrelated context.
The failure mode that surprises teams is candidate starvation: a fast lexical stage returns ten near-duplicates, leaving no room for a relevant paraphrase. Track source diversity and deduplicate before reranking. A slower second tier cannot recover a document that the first tier never admitted.
That is a data-shape problem.
There is a real trade-off here. Three tiers create more metrics, thresholds, and privacy review work. A tiny corpus with a generous deadline may be better served by one local pass. Stick with the simpler pipeline when the operational cost of tuning tiers exceeds the relevance gain you can demonstrate on the golden set.
Top comments (0)