RAG

Enterprise RAG Architecture: What Actually Breaks in Production

Permission-aware retrieval, citation grounding, and evaluation loops — the three things that separate a RAG demo from a system a company can rely on.

By Naeem Akhtar · 10 min read

In short

  • An enterprise RAG system is defined less by retrieval plus generation than by three constraints: access control, source traceability and measurable correctness.
  • Permission checks belong in the retrieval query, before any content reaches the model; instructions in a prompt are not access control.
  • Citations make answers verifiable and debugging faster, because you can see exactly what the model saw.
  • A standing evaluation set run on every meaningful change is the only dependable way to catch confident, wrong answers before users do.
01

A RAG demo and an enterprise RAG system solve different problems

A weekend RAG demo answers questions from a handful of PDFs. An enterprise RAG system has to answer questions correctly while respecting who's allowed to see what, cite where the answer came from, and degrade gracefully when the retrieved context doesn't actually contain the answer. Almost none of the difficulty is in the retrieval-plus-generation loop itself — it's in those three constraints. For implementation detail, see Building a Production RAG System with PostgreSQL + pgvector and our RAG knowledge systems service.

02

Permission-aware retrieval is not optional for internal knowledge

The moment a knowledge system spans more than one team's documents, retrieval has to respect the same access controls as the source systems — otherwise you've built a very efficient way to leak HR documents to engineering, or client contracts to the wrong account team. That means permission checks need to happen at the retrieval layer, before anything reaches the model, not as a prompt instruction telling the model to "be careful." Models don't reliably enforce access control from instructions alone.

03

Every answer should be traceable back to a source

If a system can't show which document a claim came from, users have no way to verify it — and eventually they stop trusting it entirely, which defeats the purpose of building it. Citation isn't a UI nicety; it's the mechanism that makes the system verifiable, and it also makes debugging bad answers dramatically faster, because you can see exactly what the model was working from.

04

Build the evaluation loop before you need it

The failure mode that actually damages trust in an enterprise RAG system isn't a rare weird answer — it's a confident, well-formatted, wrong one. The only reliable way to catch this before users do is a standing evaluation set: real questions with known-good answers, run against the system on every meaningful change, checked for both correctness and proper grounding in retrieved sources. Teams that skip this find out about drift from an angry Slack message instead.

05

Reference architecture for enterprise RAG

LayerWhat it doesEnterprise requirement
Source connectorsPull documents from drives, wikis, tickets and databasesIncremental sync, deletion handling, source-of-truth identifiers
Parsing and chunkingTurn files into searchable passages with metadataPreserve tables, headings and document structure; store the source and section
Access control metadataAttach who may see each chunkMirror source-system permissions; refresh on change
Index and retrievalHybrid keyword and vector search with filtersPermission and tenant filters applied inside the query
Re-rankingOrder the top candidates by relevanceKeeps the best passage in the prompt window
Generation with citationsCompose an answer grounded in retrieved passagesCite sources, refuse when evidence is missing
Evaluation and monitoringScore quality on known questions and live trafficRegression checks before release, alerting on drift
06

Failure modes that appear after launch

FailureTypical causeFix
Confident wrong answerRetrieved passages don't contain the answer; model fills the gapRequire citations; instruct and test refusal; add an 'insufficient evidence' path
Missing obvious documentPoor chunking, filter applied after top-k, or stale indexStructure-aware chunking, filters inside the search, freshness monitoring
Data leakage across teamsAccess control applied in the prompt or after retrievalEnforce permissions in the query; test with users who should see different results
Outdated answersSuperseded documents still indexedVersion metadata, deletion sync, prefer latest
Right documents, wrong answerConflicting sources or unstructured tablesSource ranking, table-aware parsing, show conflicting citations
Slow responsesLarge candidate sets, no caching, heavy re-rankingTune top-k, cache frequent queries, measure latency per stage
07

What to put in an evaluation set

Question typeExampleWhat it checks
Direct lookupWhat is the notice period in the standard services agreement?Basic retrieval and extraction
Multi-documentWhich of our suppliers hold both certifications?Combining evidence from several sources
UnanswerableA question the corpus has no information onRefusal instead of invention
PermissionedA question only answerable from a restricted document, asked as a user without accessAccess control
Superseded contentA question where an older and a newer document disagreeRecency handling
AmbiguousA vague question that needs clarificationAsking instead of guessing

For build-versus-buy considerations, see our guide to RAG knowledge base tools, and for a concrete Postgres implementation, our production RAG with pgvector article.

08

Frequently asked questions

What is enterprise RAG?

Enterprise RAG is retrieval-augmented generation built to business standards: it respects document permissions, cites its sources, handles missing evidence safely and is continuously evaluated. It differs from a demo mainly in those constraints rather than in the basic retrieve-then-generate loop.

How do you enforce permissions in a RAG system?

Store access metadata with each chunk and apply the permission filter inside the retrieval query so unauthorised content never reaches the model. Prompt instructions alone are not reliable access control.

How do you reduce hallucinations in RAG?

Improve retrieval quality, require the model to cite retrieved passages, instruct and test it to say when the evidence is insufficient, and run a standing evaluation set on every change so regressions are caught early.

How often should a RAG evaluation set be run?

Run it automatically on every change to prompts, models, chunking or retrieval settings, and on a schedule against the live system to detect drift as the underlying documents change.

Work with Eveonix

Need help with RAG and Knowledge Systems? Talk to the engineers.

Tell us what you want to build. You'll get a straight answer on scope, approach and cost from the people who do the work.

In production

See this in production