Short answer: use vector search for the claimant's supplied records, web search only for current public rules, and make every retrieved item produce an append-only audit event with a citation, timestamp, and decision context.
Keep it append-only.
That split sounds tidy until a claims adjuster asks, “Which sentence caused this recommendation?” A similarity score is not an answer. An audit trail has to preserve the exact source bytes, retrieval query, model inputs, and the policy version that was in force at the time.
Why a single index fails an insurance claim
Claims intake mixes two very different evidence streams. The FNOL form, repair estimate, photographs, and prior correspondence belong to the claim. They are private, bounded, and usually stable after ingestion. A state filing requirement or a regulator bulletin is public and can change between two calls.
Putting both streams into one vector index creates an attribution problem. A chunk from an old bulletin can rank above the current filing rule, while a claimant's phrase such as “water damage” can retrieve a semantically similar exclusion from the wrong product line. The system may sound confident because the language model sees coherent text; the log still cannot prove that the text was applicable.
The practical contract is therefore two-stage: retrieve claim-local evidence with vectors, then resolve explicitly named public authorities with a freshness-aware web path. Keep the result sets separate in storage and in the audit record. A downstream prompt can combine them, but it should never erase their provenance.
How should vector and web retrieval share an audit contract?
Start with an event schema before choosing an embedding model. For each retrieval, record a request identifier, claim identifier, actor, normalized query, source class, document or URL identifier, content hash, chunk boundaries, rank, score, retrieval timestamp, and policy snapshot. Store the citation span that was actually shown to the model, not only a document title.
Here is a small Python shape I use in an evaluation harness. It is deliberately boring: JSON Lines is easy to diff, stream, and load into a warehouse.
from dataclasses import asdict, dataclass
from datetime import datetime, timezone
import hashlib
import json
@dataclass
class RetrievalEvent:
request_id: str
claim_id: str
source_class: str # claim_document or public_web
source_id: str
content_hash: str
query: str
rank: int
score: float | None
retrieved_at: str
citation: str
policy_version: str
def hash_text(text: str) -> str:
return hashlib.sha256(text.encode("utf-8")).hexdigest()
def to_jsonl(event: RetrievalEvent) -> str:
return json.dumps(asdict(event), sort_keys=True)
event = RetrievalEvent(
request_id="req-2026-0042",
claim_id="clm-781",
source_class="claim_document",
source_id="estimate.pdf#page=2",
content_hash=hash_text("roof repair estimate ..."),
query="roof repair estimate covered peril",
rank=1,
score=0.87,
retrieved_at=datetime.now(timezone.utc).isoformat(),
citation="estimate.pdf, page 2, lines 14-22",
policy_version="home-2026-01",
)
print(to_jsonl(event))
For web results, score can be null because the provider may return ordering without a comparable similarity metric. Capture the canonical URL, response timestamp, and a content hash instead. If the page is unavailable later, the hash and retained excerpt let an auditor detect that the evidence changed; they do not pretend the old page is still current.
The failure modes worth testing before production
The first trap is citation drift. A pipeline chunks a PDF, then silently re-chunks it after an OCR change. The same citation label now points to different text. Hash the normalized chunk and include offsets so a replay can detect this. In one evaluation pass, imagine an estimate whose page numbering changes after a rescanned signature is inserted: a reviewer clicking “page 2” lands on a new paragraph, while the model's answer still quotes the old one. Without the hash, both records look legitimate. With it, the mismatch becomes a concrete review task: restore the retained excerpt, mark the index version, and rerun the claim-level test before releasing a decision.
The second trap is query leakage. A web query that includes a claimant's name or policy number can expose personal data to an external search service. Redact identifiers and log the redaction decision; “web search” is a routing choice, not permission to export the whole claim.
The third trap is stale authority. A result can be factually correct and still invalid for the claim's loss date. Store effective dates when the source provides them, and make the adjudication prompt ask for an explicit “unknown” when date applicability cannot be established.
The fourth trap is ungrounded synthesis. Retrieval metrics such as recall@k do not prove that the generated answer cited the right clause. Build a small labeled set of claims with expected supporting spans, then score citation precision and temporal validity separately. A high cosine score is not proof of grounding. A 0.87 match to the wrong endorsement is still wrong.
A compact decision rule for the intake pipeline
Use vectors when the question is about material already attached to the claim: “Does the estimate mention mold remediation?” Use web retrieval when the question names an external authority or asks for a current requirement: “What notice period does the state bulletin specify?” If both are needed, run them as independent branches and join on the request identifier.
Do not force web search into a closed-world claim review. It adds an external dependency and a new privacy boundary. Conversely, do not freeze public rules into the claim index and call that freshness. The catch is operational: retaining page snapshots and hashes costs storage, and some sites disallow automated access. When retention or access terms make that unacceptable, stick with an approved regulatory feed or a human verification step.
Before copying this pattern, measure three things on your own claims: citation precision at the span level, the percentage of answers whose sources were effective on the loss date, and replay agreement after re-indexing. Track p95 retrieval latency too, but do not let latency hide an audit gap.
References
- https://arxiv.org/abs/2005.11401
- https://www.nist.gov/itl/ai-risk-management-framework
- https://www.w3.org/TR/prov-overview/
Top comments (0)