RAG Systems

RAG combines the power of LLMs with external knowledge retrieval. Instead of relying solely on what the model learned during training, RAG retrieves relevant documents and includes them in the prompt. This reduces hallucinations and enables working with private or current information.

Where It Is Used

RAG powers knowledge-intensive applications across industries:

• Enterprise search: Internal documentation, wikis, knowledge bases
• Customer support: FAQ retrieval, support ticket resolution
• Legal: Case law search, contract analysis, compliance checking
• Healthcare: Medical literature search, clinical decision support
• Finance: Research reports, regulatory documents, news
• E-commerce: Product information, review summarization

Products: Perplexity, Bing Chat, enterprise copilots, documentation assistants.

Typical Data You Will See

Document types: - PDF, Word, HTML, Markdown - Structured data (tables, CSVs) - Code repositories - Conversation logs - Images with descriptions (multimodal)

What You Do in Practice

1. Define scope - What questions should the system answer?
2. Collect documents - Gather all relevant knowledge sources
3. Chunk documents - Split into retrieval-friendly pieces
4. Generate embeddings - Create vector representations
5. Build index - Store vectors in searchable database
6. Implement retrieval - Query pipeline with reranking
7. Design prompts - How to include retrieved context
8. Evaluate quality - Retrieval accuracy, answer quality
9. Handle failures - No results, poor results, contradictions
10. Monitor and iterate - Track usage, improve retrieval

Baseline First

Start with simple naive RAG before optimization:

# Simple RAG pipeline
query_embedding = embed(user_query)
chunks = vector_db.search(query_embedding, top_k=5)
context = "\n\n".join(chunks)

response = llm.generate(
    f"Context:\n{context}\n\nQuestion: {user_query}\nAnswer:"
)

Why naive RAG baseline: - Validates data pipeline works - Establishes quality floor - Identifies obvious gaps - Fast to implement

Only add complexity (reranking, query expansion, hybrid search) when baseline limitations are clear.

RAG Architecture Components

Typical architecture:

Query → Query processing → Retrieval → Reranking → 
Context assembly → LLM generation → Post-processing → Response

Chunking Strategies

Fixed-size chunking:

chunks = split_text(doc, chunk_size=500, overlap=50)

• Simple, predictable
• May split mid-sentence or mid-concept

Semantic chunking: - Split on paragraphs, sections, or semantic boundaries - Preserves meaning better - More complex implementation

Recursive chunking: - Start with large chunks, recursively split if too big - Respects document structure - Good for varied document types

Metadata-aware chunking: - Keep headers, document titles, dates with chunks - Improves retrieval context - Enables filtering

Best practices: - Include overlap (10-20%) between chunks - Keep chunk size appropriate for context window - Test different sizes on your data - Include metadata for filtering

Evaluation and Metrics

RAG evaluation requires measuring both retrieval and generation:

Evaluation framework: 1. Create golden dataset (50-200 Q&A pairs) 2. Measure retrieval quality independently 3. Measure generation quality with perfect retrieval 4. Measure end-to-end quality 5. Identify whether failures are retrieval or generation

Tools: RAGAS, TruLens, custom evaluation pipelines.

Failure Modes and Debugging Checklist

• [ ] Retrieval failures: Relevant docs not retrieved
• [ ] Chunk boundary issues: Answer spans multiple chunks
• [ ] Outdated embeddings: Index not updated with new docs
• [ ] Query-document mismatch: User phrasing differs from docs
• [ ] Context overflow: Too many chunks, truncation
• [ ] Hallucination despite context: LLM ignores retrieved info
• [ ] Contradictory chunks: Different docs say different things
• [ ] Missing metadata: Cannot filter to relevant subset

Production Checklist

Before deployment: - [ ] Chunking tested on representative documents - [ ] Embedding quality validated - [ ] Retrieval evaluated on test queries - [ ] Context assembly handles edge cases - [ ] Fallback for no results - [ ] Content filtering in place

After deployment: - [ ] Retrieval latency monitored (p50, p99) - [ ] Query patterns analyzed - [ ] Failed queries logged and reviewed - [ ] Index freshness maintained - [ ] Embedding model versioning tracked - [ ] Regular quality sampling

Scaling considerations: - [ ] Vector index sharding for large corpora - [ ] Embedding caching for repeated chunks - [ ] Query rate limiting - [ ] Index update strategy (batch vs streaming)

Advanced Techniques

Query expansion: - Generate multiple query variations - Use LLM to rephrase queries - Improves recall

Hybrid search:

# Combine vector and keyword search
vector_results = vector_search(query_embedding)
keyword_results = bm25_search(query_text)
merged = reciprocal_rank_fusion(vector_results, keyword_results)

Multi-stage retrieval: 1. Fast vector search (top 100) 2. Cross-encoder reranking (top 10) 3. Context assembly

Parent document retrieval: - Index small chunks for precision - Return larger parent chunks for context

Self-querying: - LLM extracts filters from natural language - "Show me contracts from 2023" → date_filter: 2023

Tools and Stack

Example Mini-Case

Scenario: Law firm needs AI assistant for searching case law and regulations across 500K documents.

Approach: 1. Document processing: PDF → text with section extraction 2. Chunking: 400 tokens with 50 token overlap, preserve section headers 3. Embeddings: OpenAI text-embedding-3-small 4. Index: Pinecone with metadata (date, court, topic) 5. Retrieval: Hybrid (vector + BM25), rerank top 20 → top 5 6. Generation: GPT-4 with citations

Metrics achieved: - Retrieval precision@5: 82% - Answer correctness: 78% - Latency p50: 1.8s, p99: 4.2s - Cost per query: $0.08

Key decisions: - Metadata filtering critical for date/jurisdiction - Reranking improved precision by 15% - Citation requirement reduced hallucinations - Fallback to search results when confidence low

Related Pages

• Index: ML Field Guide
• LLM Integration Patterns
• Semantic Search Systems
• Text Classification at Scale

Further Reading

• LlamaIndex Documentation
• Pinecone Learning Center
• RAGAS Evaluation Framework
• Building Production-Ready RAG Applications

Try it live

Everything above runs in your browser — open Vector Database Embedding Explorer and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.