RAG Permissions at Retrieval Time: Why \\\\\\\"Don't Show It\\\\\\\" Fails as Agent Access Control

Tutorial RAG systems embed a folder and return the top five chunks. Production RAG systems embed a company with HR files, board decks, customer contracts, and incident reports where half the pages are restricted to specific teams. The moment multiple users with different permissions query the same index, you have an authorization problem that prompt instructions cannot solve.

The most common mistake is trying to enforce access control in the prompt:

You may only use documents the user is permitted to see. 
Do not reveal confidential information.

By the time the model reads that instruction, the confidential chunk is already in its context window. You are asking a probabilistic text generator to un-know something you just handed it. Sometimes it complies. Sometimes a well-phrased follow-up or a prompt injection inside another document extracts it anyway.

Where Post-Hoc Filtering Breaks

Failure 1: Filtering After Retrieval

A pattern that looks reasonable:

chunks = vector_store.similarity_search(query, k=20)
allowed = [c for c in chunks if user_can_read(user, c.metadata["doc_id"])]
context = allowed[:5]

This has two problems. First, if the filter has a bug or a missing branch, the fallback is "everything." Second, top-k is computed over documents the user cannot see. A restricted folder full of near matches can push every allowed result out of the window, leaving the user with an empty or degraded answer.

Failure 2: Metadata Leakage in Hybrid Search

Hybrid search systems merge vector similarity results with keyword BM25 results. If the vector store filters by ACL but the keyword index does not, the merge step can leak document titles, folder names, or snippet previews from forbidden documents. The LLM never sees the full text, but it sees enough metadata to infer structure or content.

Failure 3: Re-Ranking Without Re-Authorization

Many RAG pipelines retrieve 50 chunks, then use a cross-encoder or LLM to re-rank them down to 5. If the re-ranker runs after the ACL filter, you are fine. If it runs before, the re-ranker sees forbidden documents and may promote them based on relevance. The final filter then removes them, but the ranking signal has already been contaminated.

Retrieval-Time Access Control Architecture

Production systems enforce permissions during the similarity search, not after. This requires injecting user context into the query itself.

Option 1: Metadata Filters in the Vector Store

Most vector databases support metadata filters that run alongside the similarity search:

chunks = vector_store.similarity_search(
    query,
    k=5,
    filter={"$or": [
        {"owner_id": user.id},
        {"shared_with": {"$contains": user.id}},
        {"visibility": "public"}
    ]}
)

This works when ACLs are simple (owner, shared list, public flag). It breaks down when permissions are computed dynamically (group membership, role hierarchies, time-based grants). The vector store has no way to evaluate complex authorization logic.

Option 2: Separate Indexes Per Tenant

For multi-tenant SaaS, some teams maintain one vector index per customer. Retrieval is scoped to the tenant's index by default. This eliminates cross-tenant leakage but creates operational overhead. You need to manage index lifecycle (create, update, delete), handle schema migrations across thousands of indexes, and route queries to the correct index based on authentication context.

Option 3: ACL-Aware Embeddings

Embed the document along with its access control metadata. Store (embedding, doc_id, acl_hash) in the vector store. At query time, compute the user's effective ACL hash and filter results to matching hashes. This works when ACLs are stable and coarse-grained (department-level, project-level). It fails when permissions change frequently or are user-specific, because you would need to re-embed and re-index every affected document.

Option 4: Authorization Proxy Layer

Place an authorization service between the RAG pipeline and the vector store. The proxy intercepts every retrieval request, fetches the user's effective permissions from an external policy engine (OPA, Zanzibar, custom RBAC), and rewrites the vector query to include the appropriate filters. This decouples permission logic from the vector store and the LLM orchestration layer, but it adds latency and a new failure mode.

Performance Trade-Offs

State Management and Cache Invalidation

When permissions change, the retrieval layer needs to know immediately. If a user is removed from a project, their next query must not return documents from that project. This requires either:

• Synchronous permission checks on every retrieval (high latency, high load on the authorization service)
• Short-lived permission caches with aggressive TTLs (5-60 seconds)
• Event-driven cache invalidation where the authorization service pushes updates to the RAG layer when ACLs change

Most production systems use a hybrid: cache permissions for 30 seconds, but subscribe to a permission-change event stream to invalidate specific user caches immediately.

Observability Gaps

Standard RAG observability tracks query latency, chunk relevance, and LLM token usage. It does not track:

• How many chunks were filtered out due to permissions
• Whether the user received a degraded answer because all relevant chunks were forbidden
• Whether a query returned zero results due to an ACL misconfiguration versus genuinely missing data

You need custom metrics:

metrics.record("rag.chunks.retrieved", len(all_chunks))
metrics.record("rag.chunks.allowed", len(allowed_chunks))
metrics.record("rag.chunks.filtered", len(all_chunks) - len(allowed_chunks))

If chunks.filtered is consistently high for a user or query pattern, either the ACLs are too restrictive or the index contains too many documents the user cannot see. Both are actionable signals.

Failure Modes

Silent degradation. The user asks a question. The answer is technically correct but incomplete because the most relevant chunks were filtered out. The LLM has no way to know it is missing context, so it does not tell the user. The user assumes the answer is complete.

ACL drift. Permissions are updated in the source system (Google Drive, Notion, Confluence) but not propagated to the vector store metadata. The RAG system continues to enforce stale permissions, either over-restricting (user cannot see documents they should) or under-restricting (user sees documents they should not).

Metadata injection. An attacker uploads a document with crafted metadata ({"shared_with": ["*"]}). If the ingestion pipeline does not validate metadata against the source system's actual ACLs, the document becomes visible to everyone.

Technical Verdict

Use retrieval-time filtering (metadata filters or separate indexes) when:

• You have more than one user with different permissions querying the same corpus
• Your ACLs are stable and can be represented as metadata tags or tenant IDs
• You can tolerate 30-second permission propagation delays

Avoid post-hoc filtering (prompt instructions or post-retrieval ACL checks) when:

• You are building a multi-tenant SaaS product
• Your users have different security clearances or data access levels
• A permission violation would violate compliance requirements (HIPAA, SOC 2, GDPR)

Use an authorization proxy when:

• Permissions are computed dynamically (role hierarchies, attribute-based access control)
• You need sub-second permission propagation
• You already have a centralized policy engine (OPA, Zanzibar)

Do not rely on the LLM to enforce access control. By the time the model sees the instruction, the access decision has already been made.

Source Links

• Original article on Dev.to