Enterprise RAG Architecture: What Actually Breaks in Production
Permission-aware retrieval, citation grounding, and evaluation loops — the three things that separate a RAG demo from a system a company can rely on.
By Naeem Akhtar · 7 min read
A RAG demo and an enterprise RAG system solve different problems
A weekend RAG demo answers questions from a handful of PDFs. An enterprise RAG system has to answer questions correctly while respecting who's allowed to see what, cite where the answer came from, and degrade gracefully when the retrieved context doesn't actually contain the answer. Almost none of the difficulty is in the retrieval-plus-generation loop itself — it's in those three constraints.
Permission-aware retrieval is not optional for internal knowledge
The moment a knowledge system spans more than one team's documents, retrieval has to respect the same access controls as the source systems — otherwise you've built a very efficient way to leak HR documents to engineering, or client contracts to the wrong account team. That means permission checks need to happen at the retrieval layer, before anything reaches the model, not as a prompt instruction telling the model to "be careful." Models don't reliably enforce access control from instructions alone.
Every answer should be traceable back to a source
If a system can't show which document a claim came from, users have no way to verify it — and eventually they stop trusting it entirely, which defeats the purpose of building it. Citation isn't a UI nicety; it's the mechanism that makes the system verifiable, and it also makes debugging bad answers dramatically faster, because you can see exactly what the model was working from.
Build the evaluation loop before you need it
The failure mode that actually damages trust in an enterprise RAG system isn't a rare weird answer — it's a confident, well-formatted, wrong one. The only reliable way to catch this before users do is a standing evaluation set: real questions with known-good answers, run against the system on every meaningful change, checked for both correctness and proper grounding in retrieved sources. Teams that skip this find out about drift from an angry Slack message instead.