Retrieval-Augmented Generation

RAG (Retrieval-Augmented Generation) is a technique that grounds LLM responses in retrieved documents. Given a query, relevant content is fetched from a knowledge base and injected into the prompt before generation. This gives the LLM access to current, specific, or proprietary information beyond its training data.

Imagine you're answering a question but you can only use what you've memorized. You might get it right, or you might confidently remember something wrong. Now imagine you can first look things up in a library before answering. You search for relevant pages, read them, and then give your answer based on what you found — plus what you already know. RAG gives AI models that ability. Instead of relying only on what the model learned during training (which has a cutoff date and may be wrong), RAG first retrieves relevant documents from a knowledge base, then uses those documents to inform the response. This makes AI systems more accurate for specific topics, keeps them up to date, and lets them work with your own private data that was never in their training.

The problem LLMs have two weaknesses for factual tasks: 1. Their knowledge is frozen at training time — they don't know about recent events 2. They can hallucinate — generate plausible-sounding but incorrect information What RAG does RAG solves this by giving the LLM relevant documents to read before answering. Instead of relying only on what it memorized, the model can consult a knowledge base at inference time. The pipeline, step by step Step 1 — Ingestion (done once, offline): Documents (PDFs, web pages, database records) are split into chunks, converted to vector embeddings (numbers that capture semantic meaning), and stored in a vector database. Step 2 — Retrieval (at query time): The user's question is converted to a vector embedding. The vector database finds the most semantically similar chunks. Step 3 — Augmentation: The retrieved chunks are injected into the LLM prompt: "Here are some relevant documents: [chunks]. Using this information, answer: [question]" Step 4 — Generation: The LLM reads the context and generates an answer grounded in the retrieved documents. A concrete example You ask: "What were last quarter's earnings?" Without RAG: The LLM has no idea — this happened after its training cutoff. With RAG: The system retrieves the earnings report from your document store, injects it into the prompt, and the LLM synthesizes the answer from the actual document.

Vector embeddings and semantic search Embeddings are dense numeric representations of text in high-dimensional space (typically 768–3072 dimensions), generated by an embedding model (e.g., OpenAI text-embedding-3-large, Nomic Embed, BGE). Semantically similar texts have similar vector representations (small cosine distance). This enables semantic search: finding conceptually relevant chunks even when they share no exact keywords with the query. Vector databases Specialized databases optimized for approximate nearest-neighbor (ANN) search: Pinecone, Weaviate, Qdrant, Chroma, pgvector. They index vectors for efficient similarity search at scale. Chunking strategies How you split documents affects retrieval quality: - Fixed-size: Simple, predictable, loses context at boundaries - Sentence/paragraph-aware: Preserves semantic units - Semantic chunking: Split where content changes topic (using embedding distance) - Hierarchical: Store both fine chunks (for precision) and parent chunks (for context) Retrieval enhancements - Hybrid search: Combine semantic (vector) with keyword (BM25) search. Often outperforms either alone, especially for proper nouns, codes, and IDs - Reranking: A cross-encoder model re-scores retrieved chunks for higher precision (e.g., Cohere Rerank, BGE Reranker) - HyDE (Hypothetical Document Embeddings): Generate a hypothetical answer to the query, embed it, use that for retrieval - Multi-query retrieval: Generate multiple query variants, retrieve for each, deduplicate Evaluation RAG quality decomposes into: - Retrieval recall: Are the right chunks being retrieved? - Retrieval precision: Are retrieved chunks relevant? - Generation faithfulness: Does the answer stay grounded in retrieved content? - Answer relevance: Does the answer actually address the question? Frameworks: RAGAS, TruLens, DeepEval.

Architectural variants - Naive RAG: Basic retrieve-then-generate. Good baseline; brittle for complex queries - Advanced RAG: Pre-retrieval (query transformation) + post-retrieval (reranking, context compression) - Modular RAG: Swappable retriever, reranker, generator components - Agentic RAG: The retrieval step is itself an agent action; the model decides when and what to retrieve Context window as a bottleneck Retrieved chunks compete for context window space with system prompt, conversation history, and output. Strategies: - Rank and truncate: Only include top-k chunks that fit - Context compression (LLMLingua, RECOMP): Compress retrieved content before injection - Iterative retrieval: Multi-turn retrieval where each retrieval is informed by prior results Production reliability - Fallback behavior: What happens when retrieval returns nothing relevant? Define a graceful degradation path - Staleness management: When retrieved documents are updated, re-embed and re-index them - Citation and provenance: Surface source URLs alongside answers for verification and trust - Latency budget: Embedding the query + ANN search + reranking adds latency. Profile and cache embedding for repeated queries Security considerations - Retrieval can surface sensitive documents to unauthorized users if access control is not enforced at the retrieval layer. Filter retrieved results by user permissions - Prompt injection risk: a malicious document in the knowledge base could inject instructions that redirect LLM behavior - Do not include credentials, PII, or confidential data in the vector index unless the retrieval system enforces row-level access control When RAG vs. fine-tuning - RAG: best for factual grounding, dynamic/proprietary knowledge, avoiding hallucination on specific facts - Fine-tuning: best for style, tone, domain-specific reasoning patterns, reducing instruction overhead - They are not mutually exclusive — fine-tuned models with RAG is a common production pattern Cost model Costs: embedding model calls (ingestion) + embedding at query time + vector DB storage + ANN search + LLM generation. At scale, embedding ingestion and storage dominate; optimize by chunking efficiently and using dimension reduction where quality allows.