Semantic Search Systems

Semantic search finds documents by meaning, not just keywords. Instead of exact term matching, it uses embeddings to represent queries and documents as vectors, then finds similar vectors. This enables "find documents about machine learning" to match "artificial intelligence tutorial" even without shared words.

Where It Is Used

Semantic search powers modern information retrieval:

• Enterprise search: Internal documents, wikis, Slack messages
• E-commerce: Product discovery beyond keyword matching
• Support: Finding similar tickets, FAQ matching
• Legal: Case law and document similarity
• Healthcare: Medical literature search
• Code search: Finding similar code snippets

Products: Google Search, Elasticsearch with vectors, Algolia, Pinecone, Weaviate.

Typical Data You Will See

Data sources: - Text documents (PDF, HTML, markdown) - Product catalogs with descriptions - Support tickets and conversations - Code repositories - Structured data with text fields

What You Do in Practice

1. Define scope - What are users searching for? What should match?
2. Choose embedding model - Balance quality, speed, and cost
3. Preprocess documents - Clean, chunk, and prepare text
4. Generate embeddings - Batch process document corpus
5. Build index - Choose vector database, configure parameters
6. Implement query pipeline - Query processing, search, post-processing
7. Add hybrid search - Combine semantic with keyword search
8. Evaluate and tune - Measure relevance, adjust parameters
9. Monitor production - Track latency, relevance, and usage

Baseline First

Start with BM25 keyword search before embeddings:

from elasticsearch import Elasticsearch

es = Elasticsearch()

# BM25 baseline
response = es.search(
    index="documents",
    body={
        "query": {
            "match": {
                "content": user_query
            }
        }
    }
)

Why keyword baseline: - Fast and well-understood - Works well for exact term queries - Establishes relevance floor - Identifies where semantic adds value

Move to semantic when: - Users use varied vocabulary - Synonym and concept matching needed - Keywords alone miss relevant results

Embedding Models

Selection criteria: - Quality: How well does it capture semantic similarity? - Speed: Latency for real-time queries - Cost: API costs vs self-hosting - Privacy: Can data leave your infrastructure?

Example with sentence-transformers:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L6-v2')

# Encode documents (batch)
doc_embeddings = model.encode(documents, batch_size=32)

# Encode query (single)
query_embedding = model.encode(query)

# Similarity
from sklearn.metrics.pairwise import cosine_similarity
similarities = cosine_similarity([query_embedding], doc_embeddings)[0]

Vector Search Architecture

Components:

Documents → Chunking → Embedding → Vector Index
                                        ↓
Query → Query Embedding → ANN Search → Top-K → Reranking → Results

Vector databases:

Index types: - HNSW: Best quality, higher memory - IVF: Good balance, tunable - Flat: Exact search, slow for large corpora

Hybrid Search

Combine semantic and keyword search for best results:

def hybrid_search(query, alpha=0.7):
    """Combine BM25 and vector search."""
    # Keyword search
    bm25_results = keyword_search(query)
    bm25_scores = normalize(bm25_results)

    # Semantic search
    query_embedding = embed(query)
    vector_results = vector_search(query_embedding)
    vector_scores = normalize(vector_results)

    # Combine with reciprocal rank fusion
    combined = reciprocal_rank_fusion(
        [bm25_results, vector_results],
        weights=[1-alpha, alpha]
    )

    return combined

def reciprocal_rank_fusion(result_lists, weights=None, k=60):
    """Merge ranked lists using RRF."""
    scores = defaultdict(float)
    for i, results in enumerate(result_lists):
        weight = weights[i] if weights else 1
        for rank, doc_id in enumerate(results):
            scores[doc_id] += weight * (1 / (k + rank + 1))
    return sorted(scores.items(), key=lambda x: -x[1])

When to use hybrid: - Product names and specific terms (keyword helps) - Conceptual queries (semantic helps) - Mixed query types in production

Evaluation and Metrics

Evaluation process: 1. Create test set (100-500 queries with relevance labels) 2. Run queries through system 3. Compute metrics at k=5, 10, 20 4. Compare against baselines 5. Analyze failure cases

Tools: - BEIR benchmark for embedding comparison - MTEB leaderboard for model selection - Custom evaluation on your data

Failure Modes and Debugging Checklist

• [ ] Embedding quality: Model doesn't capture domain semantics
• [ ] Chunking issues: Relevant info split across chunks
• [ ] Query-document mismatch: Queries phrased differently than docs
• [ ] Stale index: New documents not indexed
• [ ] Latency spikes: Index not optimized
• [ ] Recall problems: Top-k too small, wrong index parameters
• [ ] Keyword dependency: Exact terms missed by semantic

Production Checklist

Before deployment: - [ ] Embedding model tested on domain data - [ ] Chunking strategy validated - [ ] Index parameters tuned (ef, m for HNSW) - [ ] Hybrid search balance calibrated - [ ] Latency tested at expected load - [ ] Fallback for search failures

After deployment: - [ ] Query latency monitored (p50, p99) - [ ] Relevance feedback collected - [ ] Index freshness maintained - [ ] No-result queries analyzed - [ ] Embedding model version tracked

Scaling considerations: - [ ] Batch embedding generation - [ ] Query embedding caching for repeated queries - [ ] Index sharding for large corpora - [ ] Replication for high availability

Reranking

Two-stage architecture for quality:

def search_with_reranking(query, top_k=10, rerank_k=100):
    """Fast retrieval then precise reranking."""
    # Stage 1: Fast vector search
    candidates = vector_search(query, k=rerank_k)

    # Stage 2: Cross-encoder reranking
    pairs = [(query, doc.text) for doc in candidates]
    rerank_scores = cross_encoder.predict(pairs)

    # Combine and sort
    reranked = sorted(
        zip(candidates, rerank_scores),
        key=lambda x: -x[1]
    )[:top_k]

    return reranked

Reranking models: - Cross-encoders (more accurate, slower) - Cohere Rerank API - BGE Reranker - Custom fine-tuned models

Tools and Stack

Example Mini-Case

Scenario: Enterprise needs to search across 500K internal documents (Confluence, Google Docs, Slack).

Approach: 1. Baseline: Elasticsearch BM25 - Recall@10: 0.62 2. Chunking: 400 tokens with overlap, preserve headers 3. Embeddings: OpenAI text-embedding-3-small 4. Index: Pinecone with metadata filters 5. Hybrid: 70% semantic, 30% BM25 6. Reranking: Cohere Rerank on top 50

Results: - Recall@10: 0.62 → 0.84 (+35%) - Query latency: 180ms (within SLA) - User satisfaction: +40% in surveys

Key decisions: - Metadata filters critical for access control - Hybrid search handled product names better - Reranking worth latency cost for complex queries

Related Pages

• Index: ML Field Guide
• Learning to Rank
• RAG Systems
• Text Classification at Scale

Further Reading

• MTEB Leaderboard
• Pinecone Learning Center
• Sentence Transformers
• Vespa Hybrid Search

Try it live

Everything above runs in your browser — open Word Embeddings Explorer - NLP Vector Space Visualization and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.