Every chunk of indexed text is embedded into a high-dimensional vector; here each dot is one chunk, projected onto a unit sphere so direction alone encodes meaning. Chunks about the same topic cluster together — that's what makes nearest-neighbor search work as semantic retrieval.
Cosine similarity: sim(q,d) = (q · d) / (|q| |d|)
Hybrid rerank score: score = α·BM25(q,d) + (1-α)·sim(q,d), α ≈ 0.35
Recall@K: |relevant ∩ top-K| / |relevant|
Brute-force latency: O(N) per query · N = indexed chunks
- Chunk size — smaller chunks tighten each topic's cluster (more precise boundaries, easier to retrieve exactly) but a large corpus needs more of them; larger chunks blur cluster edges and lose recall.
- Indexed chunks — the vector store size; brute-force cosine search cost (and latency) grows roughly linearly with it, which is why production systems switch to approximate nearest-neighbor indexes (HNSW, IVF) at scale.
- Top-K retrieved — how many chunks get pulled into the context window; too few loses relevant context, too many risks "context stuffing" and higher token cost.
- Hybrid BM25 + vector — blends a simulated keyword-overlap score with cosine similarity before ranking, the same trick real hybrid retrievers use to catch exact-term matches pure embeddings miss.
Watch Recall@K and latency move in opposite directions as you push the index bigger or add hybrid reranking — that trade-off is the core engineering problem of RAG at scale.