Enterprise RAG Architecture: What Actually Breaks in Production
Permission-aware retrieval, citation grounding, and evaluation loops — the three things that separate a RAG demo from a system a company can rely on.
By Naeem Akhtar · 10 min read
In short
- An enterprise RAG system is defined less by retrieval plus generation than by three constraints: access control, source traceability and measurable correctness.
- Permission checks belong in the retrieval query, before any content reaches the model; instructions in a prompt are not access control.
- Citations make answers verifiable and debugging faster, because you can see exactly what the model saw.
- A standing evaluation set run on every meaningful change is the only dependable way to catch confident, wrong answers before users do.
A RAG demo and an enterprise RAG system solve different problems
A weekend RAG demo answers questions from a handful of PDFs. An enterprise RAG system has to answer questions correctly while respecting who's allowed to see what, cite where the answer came from, and degrade gracefully when the retrieved context doesn't actually contain the answer. Almost none of the difficulty is in the retrieval-plus-generation loop itself — it's in those three constraints. For implementation detail, see Building a Production RAG System with PostgreSQL + pgvector and our RAG knowledge systems service.
Permission-aware retrieval is not optional for internal knowledge
The moment a knowledge system spans more than one team's documents, retrieval has to respect the same access controls as the source systems — otherwise you've built a very efficient way to leak HR documents to engineering, or client contracts to the wrong account team. That means permission checks need to happen at the retrieval layer, before anything reaches the model, not as a prompt instruction telling the model to "be careful." Models don't reliably enforce access control from instructions alone.
Every answer should be traceable back to a source
If a system can't show which document a claim came from, users have no way to verify it — and eventually they stop trusting it entirely, which defeats the purpose of building it. Citation isn't a UI nicety; it's the mechanism that makes the system verifiable, and it also makes debugging bad answers dramatically faster, because you can see exactly what the model was working from.
Build the evaluation loop before you need it
The failure mode that actually damages trust in an enterprise RAG system isn't a rare weird answer — it's a confident, well-formatted, wrong one. The only reliable way to catch this before users do is a standing evaluation set: real questions with known-good answers, run against the system on every meaningful change, checked for both correctness and proper grounding in retrieved sources. Teams that skip this find out about drift from an angry Slack message instead.
Reference architecture for enterprise RAG
| Layer | What it does | Enterprise requirement |
|---|---|---|
| Source connectors | Pull documents from drives, wikis, tickets and databases | Incremental sync, deletion handling, source-of-truth identifiers |
| Parsing and chunking | Turn files into searchable passages with metadata | Preserve tables, headings and document structure; store the source and section |
| Access control metadata | Attach who may see each chunk | Mirror source-system permissions; refresh on change |
| Index and retrieval | Hybrid keyword and vector search with filters | Permission and tenant filters applied inside the query |
| Re-ranking | Order the top candidates by relevance | Keeps the best passage in the prompt window |
| Generation with citations | Compose an answer grounded in retrieved passages | Cite sources, refuse when evidence is missing |
| Evaluation and monitoring | Score quality on known questions and live traffic | Regression checks before release, alerting on drift |
Failure modes that appear after launch
| Failure | Typical cause | Fix |
|---|---|---|
| Confident wrong answer | Retrieved passages don't contain the answer; model fills the gap | Require citations; instruct and test refusal; add an 'insufficient evidence' path |
| Missing obvious document | Poor chunking, filter applied after top-k, or stale index | Structure-aware chunking, filters inside the search, freshness monitoring |
| Data leakage across teams | Access control applied in the prompt or after retrieval | Enforce permissions in the query; test with users who should see different results |
| Outdated answers | Superseded documents still indexed | Version metadata, deletion sync, prefer latest |
| Right documents, wrong answer | Conflicting sources or unstructured tables | Source ranking, table-aware parsing, show conflicting citations |
| Slow responses | Large candidate sets, no caching, heavy re-ranking | Tune top-k, cache frequent queries, measure latency per stage |
What to put in an evaluation set
| Question type | Example | What it checks |
|---|---|---|
| Direct lookup | What is the notice period in the standard services agreement? | Basic retrieval and extraction |
| Multi-document | Which of our suppliers hold both certifications? | Combining evidence from several sources |
| Unanswerable | A question the corpus has no information on | Refusal instead of invention |
| Permissioned | A question only answerable from a restricted document, asked as a user without access | Access control |
| Superseded content | A question where an older and a newer document disagree | Recency handling |
| Ambiguous | A vague question that needs clarification | Asking instead of guessing |
For build-versus-buy considerations, see our guide to RAG knowledge base tools, and for a concrete Postgres implementation, our production RAG with pgvector article.
Frequently asked questions
What is enterprise RAG?
Enterprise RAG is retrieval-augmented generation built to business standards: it respects document permissions, cites its sources, handles missing evidence safely and is continuously evaluated. It differs from a demo mainly in those constraints rather than in the basic retrieve-then-generate loop.
How do you enforce permissions in a RAG system?
Store access metadata with each chunk and apply the permission filter inside the retrieval query so unauthorised content never reaches the model. Prompt instructions alone are not reliable access control.
How do you reduce hallucinations in RAG?
Improve retrieval quality, require the model to cite retrieved passages, instruct and test it to say when the evidence is insufficient, and run a standing evaluation set on every change so regressions are caught early.
How often should a RAG evaluation set be run?
Run it automatically on every change to prompts, models, chunking or retrieval settings, and on a schedule against the live system to detect drift as the underlying documents change.
Need help with RAG and Knowledge Systems? Talk to the engineers.
Tell us what you want to build. You'll get a straight answer on scope, approach and cost from the people who do the work.
Keep reading
Related guides
RAG
Building a Production RAG System with PostgreSQL + pgvector
You probably don't need a dedicated vector database. Here's how HNSW indexing, hybrid search, and connection pooling in Postgres get RAG to production scale.
11 min read
RAG
RAG Knowledge Base Tools: Build vs Buy, and How to Choose
Off-the-shelf knowledge base tools, managed RAG services or a custom build? A comparison by data sensitivity, permissions, quality control and total cost.
11 min read
RAG
RAG vs Fine-Tuning: What Should Your Startup Actually Use?
The honest answer is usually RAG, occasionally both, and rarely fine-tuning alone — here's the decision framework and what each path actually costs to run.
10 min read