RAG Architecture: How Retrieval-Augmented Generation Actually Works

RAG — retrieval-augmented generation — grounds an LLM in specific documents at query time instead of baking them into model weights. The pipeline has five stages. First, offline indexing: split your document corpus into chunks (typically 300-800 tokens with some overlap), embed each chunk using an embedding model like text-embedding-3-small, bge, or e5, and store the embedding plus the chunk text in a vector database like pgvector, Qdrant, Milvus, or Pinecone. Chunking choices matter — too small and you lose surrounding context that the LLM needs; too large and each chunk dilutes the relevant signal. Semantic chunking that respects sentence or paragraph boundaries generally beats naive fixed-size splits. Second, online retrieval: when the user asks a question, embed the question with the same embedding model, then query the vector DB for the top-k nearest neighbors by cosine similarity. Third, optional reranking: pass the top 20-50 retrieved chunks through a cross-encoder reranker like bge-reranker or Cohere Rerank, which is a smaller model that reads the query and each chunk together and scores relevance more precisely than pure embedding similarity. Keep the top 3-5 reranked chunks. Fourth, hybrid search: in practice, combining dense embedding retrieval with sparse BM25 keyword retrieval (reciprocal rank fusion or weighted sum) catches cases where exact keywords matter — technical terms, names, numbers — that embeddings sometimes miss. Fifth, prompt assembly: construct a prompt like "Use the following context to answer. If the answer is not in the context, say you don't know. Context: [retrieved chunks]. Question: [user query]." Send to the LLM. For when to use RAG versus fine-tuning: they solve different problems. RAG injects knowledge — facts, documents, evolving domain information. Fine-tuning shapes behavior — tone, response format, following specific instructions. For the common "chat with our internal docs" use case, RAG is the right tool because documents change constantly and retraining would be expensive and produce stale results. Fine-tuning is complementary — you might fine-tune on output format while using RAG for the actual facts. The expensive failure mode to watch for: hallucination when retrieval misses. Even with "say you don't know" instructions, models sometimes confabulate. Mitigations include confidence thresholds on retrieval scores, citation requirements in the prompt, and evaluation frameworks like Ragas to monitor faithfulness in production.