Enterprise RAG: Treating Retrieval as a Security Control

The constraint

Enterprise RAG has a failure mode that consumer RAG does not: retrieving a document the user was never allowed to see. An HR document surfacing in an engineer's search result is not a relevance bug, it is a data incident.

Most tutorials retrieve first and filter second. At scale that is both slow and wrong — slow because you over-retrieve to compensate, wrong because the embedding of a restricted document has already influenced your ranking.

Approach

Permissions move into the retrieval query rather than after it. Each chunk carries its source document's ACL as vector metadata at index time. Queries are issued with a namespace filter derived from the caller's group memberships, resolved fresh per request.

The tradeoff is real and worth stating: ACL changes require reindexing affected documents. An incremental reindex pipeline triggered by document-permission change events handles this, and it is far cheaper than the alternative of runtime permission joins on every query.

Architecture

Ingestion is a Cloud Run job: extract, chunk on semantic boundaries rather than fixed token counts, embed, attach ACL metadata, upsert. Chunking on headings and paragraph breaks rather than every 512 tokens made a larger difference to answer quality than any prompt change — a chunk that ends mid-sentence retrieves badly no matter how good the embedding model is.

Retrieval is a hybrid: dense vector search for semantic recall, plus lexical BM25 for exact-match terms like error codes and product names that embeddings notoriously blur. Results are fused with reciprocal rank fusion, which needs no tuning and consistently beats either retriever alone.

Generation receives top-k chunks with citations attached, and the system prompt hard- blocks answering outside the provided context. Every response carries source document IDs so users can verify — which also surfaces retrieval failures rather than hiding them behind fluent prose.

What broke

Recall looked excellent in evaluation and users complained constantly. The cause: the golden evaluation set had been written by the team building the system, so it asked questions in the vocabulary of the documents. Real users asked in their own words.

The fix was rebuilding the evaluation set from actual logged queries. Measured recall dropped by roughly a third overnight — which was not a regression, it was the first honest number the project had.

Outcome

400k documents indexed with sub-200ms p95 retrieval. Permissions enforced at query time. Most importantly, an evaluation harness that reflects real usage, so prompt and model changes can be compared against something meaningful.