RAG Architecture for the Enterprise: Why Retrieval-Augmented Generation Needs a Governed Data Layer Behind the Vector Database

Enterprise RAG architecture is a retrieval-augmented generation design in which the retrieval step runs against a governed data layer that applies the requesting user's identity, role, and data classification before any chunk reaches the model. A vector database supplies similarity search; the governed layer supplies authorization, redaction, freshness, and an audit record for every retrieval.

Retrieval-augmented generation (RAG) is the pattern, introduced by Lewis et al. in 2020, of fetching relevant documents at query time and feeding them to a language model as context [1]. By late 2024 it was the dominant enterprise pattern: 51 percent of 600 surveyed US IT decision-makers reported RAG in production, up from 31 percent a year earlier, while only 9 percent of production models were fine-tuned [2]. In November 2024, OWASP added "Vector and Embedding Weaknesses" as entry LLM08 in its Top 10 for LLM applications, naming cross-context leakage between users who share a vector store and embedding inversion as risks specific to RAG [3].

The corpus behind an enterprise RAG system is almost always mixed: public policy documents next to HR case files, product manuals next to unreleased pricing. Retrieval therefore has to enforce the same rules as any other query against that data. The post on the 90-day path to an AI-ready data estate covers getting the data cataloged and classified; this guide covers what retrieval has to do with that classification once it exists.

Why Does Standard RAG Architecture Have No Access Control at the Retrieval Step?

Standard RAG has no access control at retrieval because the vector database knows about vectors and metadata and knows nothing about who is asking. An embedding is a fixed-length numeric vector a model produces to represent the meaning of a chunk of text, and a vector database indexes those vectors for nearest-neighbor lookup. Semantic search ranks chunks by vector distance to the query, and nothing in that ranking consults an identity provider or a sensitivity label.

The vendors document this division of labor. Pinecone partitions an index into namespaces and recommends one namespace per tenant [4]. Weaviate stores each tenant on a separate shard and requires a tenant name on every query [5]. Qdrant's multitenancy guide shows a group_id payload field and a filter on that field in every query example [6]. In each case the application decides which tenant or filter to pass, and an omission in that code path returns the full corpus.

The sibling guide on what multi-agent systems need from your data layer covers the retrieve row when the caller is an agent, and the guide on column-level security when agents query your data covers the filter row.

What Are the Three Classes of Retrieval Failure in Enterprise RAG?

Enterprise RAG fails in three distinct ways, each with its own cause and control. Over-retrieval returns content the requesting user is not authorized to see. Under-retrieval withholds content the user is authorized to see because the corpus was shrunk to stop the first failure. Index staleness returns content that was correct when embedded but has since been revised or deleted at the source.

Over-retrieval has the public track record. In March 2026, Gartner research vice-president Dennis Xu told the Security & Risk Management Summit in Sydney that "Copilot makes over-shared documents more accessible," citing an employee who asked Copilot about organizational changes and received confidential reorganization documents left readable by too broad a SharePoint group [7]. Microsoft's Restricted SharePoint Search limits Copilot to an allow list of up to 100 sites, and Microsoft describes it as "a short-term solution" that "isn't a security boundary and doesn't change any permissions" [8]. That is under-retrieval used as the mitigation for over-retrieval: shrink the corpus until the leak stops, and accept that answers degrade.

Staleness compounds the other two. Oracle's July 2026 guidance on index drift describes deleted documents that "still show up when deletion is handled in the source system but not propagated to chunk rows, embedding rows, metadata filters, and search indexes" [9]. Reclassification is the dangerous case: a document that moves from Internal to Restricted at the source stays Internal in a stale index, and a correctly written filter still over-retrieves it.

How Do You Build RAG for the Enterprise Safely With Classification-Aware Retrieval?

Safe enterprise RAG derives every retrieval filter from the caller's identity and the corpus's classification at query time, from the same policy engine that governs SQL access to the same data. Classification-aware retrieval stamps every chunk with the sensitivity class and access policy of its source document, then generates the retrieval filter from the requesting identity, with no dependence on application configuration. Apache Ranger already makes this decision for SQL, evaluating row filters and column masks per user and group with masking types that include partial reveal and NULL substitution [10]. Governed RAG applies the identical decision to chunk retrieval and to the text inside each chunk. OWASP's LLM08 mitigations call for "fine-grained access controls and permission-aware vector and embedding stores," classification tags to control access levels, and immutable logs of retrieval activity [3].

Use this checklist to test whether a design qualifies as classification-aware.

• Every source document carries a classification tag and an owner in the catalog before it is eligible for embedding.

• Every chunk stores classification, tenant, source ACL reference, source version, and source hash as metadata.

• The retrieval filter is generated per request from authenticated identity and roles by the policy engine, never hard-coded in the application.

• Filtering runs before ranking, so top-k is computed over the authorized set only.

• Masking and PII redaction rules from the policy engine apply to chunk text before it leaves the data layer.

• Reclassification or deletion at the source invalidates affected chunks within a measured interval.

• Re-ingestion uses idempotent upsert on source ID plus hash, so duplicate chunks are rejected.

• Every retrieval writes an audit record with identity, policy decision, chunk IDs, source versions, and consuming model.

• Standing challenge queries assert that known-restricted and known-retired content never appears in results.

• Embeddings of restricted content are stored only where the plaintext would be permitted to live.

The last item follows from Morris et al., who at EMNLP 2023 recovered 92 percent of 32-token inputs exactly from their embeddings and reconstructed full names from clinical notes [11]. An embedding of a restricted document is a restricted artifact and inherits the document's storage and residency constraints.

Why Not Just Use Vector Database Namespaces and Metadata Filters?

Namespaces and metadata filters are necessary and, on their own, insufficient, because they enforce a filter the application chose and the enterprise did not define. Pinecone's metadata operators narrow the candidate set before scoring [12], Qdrant's must, should, and must_not clauses nest into full boolean expressions [13], and Weaviate's per-tenant shards make cross-tenant reads impossible by construction [5]. Those are good primitives. The problem is where the filter value comes from.

In a naive design the application builds the filter from a config file or from a role lookup it maintains itself. That lookup is a second copy of the access model, and second copies drift: a user changes teams or a document is reclassified, the Ranger policy on the warehouse updates, and the RAG filter does not. Xu made the same point about SharePoint, where labels and access control lists are two overlapping controls and user error between them is the root of oversharing [7]. Namespaces also cannot express classification that varies within a tenant; a finance analyst and a finance intern share a tenant and need different retrievable sets. Keep the vector database for the filter primitives, and move the decision about their values into the governed data layer.

NexusOne Governs RAG Retrieval With the Same Policy That Governs Every Query

NexusOne treats a RAG retrieval as one more request against the estate, governed like a Trino query or a Spark job. Identity is federated from Active Directory, Okta, or SAML into one Keycloak layer. A policy defined once in Apache Ranger applies simultaneously to Trino, Apache Spark, Apache Kyuubi, and S3 object storage per object, so RAG retrieves exactly what the same user can read anywhere else. Tag a dataset in DataHub, map a role to the tag, and the policies that gate retrieval generate automatically.

The AI & Data Control Plane sits between callers and models. Its semantic router classifies each request by topic and identity, attaches the topic-scoped corpus (a finance question receives a finance RAG), routes deterministic questions to the query engine at zero tokens, and redacts PII before any request reaches a frontier model. Jailbreak and PII signals block a request with the reason logged, and every request is recorded and counted against a per-role token budget. MCP and tool calling run on the router, so agents inherit the same envelope (see the MCP trust problem). The layer runs identically on-prem, in any cloud, hybrid, or air-gapped over Apache Iceberg, the basis of SLM-first sovereign AI. To review your retrieval architecture, schedule an expert consultation.

Key Takeaways

• A vector database ranks chunks by similarity and has no knowledge of who is asking; authorization has to come from a governed data layer upstream of the search call.

• Enterprise RAG fails through over-retrieval, under-retrieval, and index staleness, and shrinking the corpus to stop the first causes the second.

• Every chunk should carry its source document's classification, tenant, ACL reference, version, and hash so the filter and the freshness check both have something to act on.

• Retrieval filters should be generated per request from the caller's identity by the same policy engine that governs SQL access, never kept as a second copy inside the AI application.

• Embeddings can be inverted to recover source text, so embeddings of restricted content inherit the storage and residency constraints of the documents they encode.

FAQ

What Does Enterprise RAG Architecture Look Like With Data Governance Built In?

Governed enterprise RAG reads from cataloged, classified sources; embeds chunks with classification, tenant, ACL, version, and hash metadata; and retrieves through a filter generated from the requesting identity by the estate's policy engine. Redaction applies to chunk text before it enters the prompt, and every retrieval is logged with identity and policy decision. The vector database remains the similarity engine, and the governed data layer decides what it may return.

What Is the Data Layer for Retrieval-Augmented Generation in the Enterprise?

The data layer for enterprise RAG is the governed access path between the RAG pipeline and the source systems: one identity model, one policy engine, a catalog with classification tags, and an audit log. It supplies the retrieval filter for each request and propagates deletions and reclassifications into the index. A vector database is one component inside that layer.

What Is a RAG Data Platform Architecture?

A RAG data platform architecture connects ingest, embedding, storage, retrieval, filtering, generation, and logging so that one governance model applies at each step. Classification is recorded at ingest and carried on every chunk, the retrieval filter comes from the policy engine, generation is routed by topic and identity with redaction before any external model call, and every step writes to a shared audit record.

How Do You Build RAG for the Enterprise Safely?

Start with classification: no document is eligible for embedding until it carries a sensitivity tag and an owner in the catalog. Derive retrieval filters from authenticated identity at request time and apply them before ranking. Propagate source deletions and reclassifications into chunk metadata on a measured interval, run standing challenge queries that confirm restricted and retired content stays absent, and log every retrieval with identity, policy decision, chunk IDs, and model.

References

1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020), arXiv, https://arxiv.org/abs/2005.11401

2. 2024: The State of Generative AI in the Enterprise, Menlo Ventures, https://menlovc.com/2024-the-state-of-generative-ai-in-the-enterprise/

3. LLM08:2025 Vector and Embedding Weaknesses, OWASP Top 10 for LLM Applications, https://genai.owasp.org/llmrisk/llm082025-vector-and-embedding-weaknesses/

4. Indexing Overview: Namespaces and Metadata, Pinecone Docs, https://docs.pinecone.io/guides/index-data/indexing-overview

5. Multi-Tenancy Operations, Weaviate Docs, https://docs.weaviate.io/weaviate/manage-collections/multi-tenancy

6. Configure Multitenancy, Qdrant Documentation, https://qdrant.tech/documentation/guides/multiple-partitions/

7. Gartner Suggests Friday Afternoon Copilot Ban (Dennis Xu, Gartner Security & Risk Management Summit, March 17, 2026), The Register, https://www.theregister.com/2026/03/17/gartner_copilot_security_mitigations/

8. Restricted SharePoint Search, Microsoft Learn, https://learn.microsoft.com/en-us/sharepoint/restricted-sharepoint-search

9. How to Detect RAG Index Drift: Deleted Docs, Stale Chunks, and Duplicate Embeddings (July 16, 2026), Oracle Developers Blog, https://blogs.oracle.com/developers/how-to-detect-rag-index-drift-deleted-docs-stale-chunks-and-duplicate-embeddings

10. Row-Level Filtering and Column-Masking Using Apache Ranger Policies in Apache Hive, Apache Software Foundation Wiki, https://cwiki.apache.org/confluence/display/RANGER/Row-level+filtering+and+column-masking+using+Apache+Ranger+policies+in+Apache+Hive

11. Text Embeddings Reveal (Almost) As Much As Text (Morris, Kuleshov, Shmatikov, Rush, EMNLP 2023), arXiv, https://arxiv.org/abs/2310.06816

12. Filter by Metadata, Pinecone Docs, https://docs.pinecone.io/guides/search/filter-by-metadata

13. Filtering, Qdrant Documentation, https://qdrant.tech/documentation/concepts/filtering/