Building Enterprise RAG Systems: Architecture Patterns That Scale

Why naive RAG fails in production

The standard RAG tutorial pattern — chunk documents into fixed-size pieces, embed them, store in a vector DB, retrieve top-k on query, pass to LLM — works well enough to demo. It fails in production for predictable reasons: fixed-size chunking breaks semantic units, top-k retrieval by cosine similarity misses context, and there is no mechanism to handle query-document mismatch when the user's vocabulary differs from the document's vocabulary.

Enterprise knowledge bases compound these problems. Internal documents use acronyms and jargon that embeddings do not handle well without fine-tuning. Policies and procedures have interdependencies that chunk-level retrieval severs. User queries in enterprise contexts are often ambiguous — 'What is the leave policy for probationary employees?' requires resolving 'probationary' to your specific company's definition before retrieval can work.

Chunking strategies that preserve semantic integrity

The chunking decision is the most underappreciated architectural choice in RAG. Fixed-size chunking (split at 512 tokens) is a fine default for tutorial demos. Production systems need chunking strategies aligned with document structure: section-based chunking for policy documents, paragraph-based chunking for narrative content, sentence-window chunking for dense technical documents, and metadata-rich chunking for structured data sources.

Hierarchical chunking is worth considering for large document corpora: create summary-level embeddings for documents and detailed-level embeddings for sections. Use the summary level to identify relevant documents, then retrieve detailed sections. This two-stage retrieval significantly improves precision on long-tail queries that would otherwise compete against too many loosely related chunks.

• Section-based chunking for structured documents (policies, procedures, contracts)
• Paragraph-based chunking for narrative content where context flows across sentences
• Sentence-window chunking: embed a sentence but retrieve its surrounding context window
• Hierarchical chunking: document summaries for coarse retrieval, sections for fine retrieval
• Preserve metadata (document title, section heading, date, author) in every chunk record

Retrieval pipeline architecture: beyond cosine similarity

Production RAG pipelines use hybrid retrieval: dense vector search (semantic similarity via embeddings) combined with sparse retrieval (keyword matching via BM25 or similar). Dense retrieval handles paraphrase and conceptual queries well but misses exact-match terms. Sparse retrieval handles exact terms and proper nouns well but misses semantic relationships. Combining them with a re-ranker on top significantly outperforms either alone.

Query transformation is another underused lever. Before retrieval, expand or rewrite the user's query: generate hypothetical document excerpts (HyDE) that would answer the query and embed those for retrieval, decompose multi-part queries into sub-queries, or use an LLM to rewrite ambiguous queries into clearer retrieval targets. The query transformation step is cheap relative to its retrieval improvement impact.

• Implement hybrid retrieval: dense (embedding) + sparse (BM25) with a re-ranking step
• Use a cross-encoder re-ranker (Cohere Rerank, BGE re-ranker) to refine top-k candidates
• Implement query transformation — HyDE, query expansion, or sub-query decomposition
• Add a metadata filtering layer — filter by document type, date, or department before semantic search
• Cache embedding results for frequently retrieved documents to reduce vector DB query load

Evaluation frameworks for RAG production quality

You cannot improve what you do not measure. Enterprise RAG systems need ongoing evaluation against ground-truth question-answer pairs from your domain. The standard evaluation triad is: context precision (are retrieved chunks relevant?), context recall (did retrieval find all relevant chunks?), and answer faithfulness (is the LLM's answer grounded in retrieved context, not hallucinated?).

Tools like RAGAS, TruLens, and LangSmith provide automated evaluation pipelines. More importantly, build a golden dataset from real user queries — the questions your users actually ask, with answers reviewed by subject-matter experts. Automated metrics improve retrieval quality; domain expert review catches failure modes that metrics miss.

• Build a golden Q&A dataset from real user queries reviewed by domain experts
• Measure context precision, context recall, and answer faithfulness as baseline metrics
• Use automated evaluation tools (RAGAS, TruLens) for continuous regression testing
• Track retrieval latency by percentile — p50, p90, p99 — not just average latency
• Implement human-in-the-loop feedback: let users flag bad answers and feed that back to retrieval improvement

Infrastructure decisions for production scale

Vector database selection matters at scale. Pinecone, Weaviate, Qdrant, and pgvector (PostgreSQL extension) have meaningfully different performance profiles, operational complexity, and cost structures. For most enterprise RAG use cases under 10 million vectors, pgvector on a well-provisioned Postgres instance is simpler to operate and cheaper than managed vector database services. Above that scale, purpose-built vector databases justify their operational overhead.

Embedding model selection is a deployment decision, not just a quality decision. Hosted embedding APIs (OpenAI, Cohere) are fast to integrate but introduce latency, cost per token, and data residency concerns. Self-hosted embedding models (BGE, E5-large) eliminate those concerns but add infrastructure to manage. For enterprise RAG with sensitive data, self-hosted embeddings and self-hosted vector storage keeps all document content on-premises.

• Under 10M vectors: evaluate pgvector before adopting a specialized vector database
• Above 10M vectors or sub-10ms latency requirements: consider Qdrant or Weaviate
• Sensitive data environments: self-host embeddings (BGE-M3, E5-large) and vector storage
• Implement async indexing — do not block document ingestion pipelines on embedding latency
• Design for incremental updates — production RAG knowledge bases change continuously