Vector Databases and Embeddings: The Infrastructure Behind Retrieval-Augmented Generation
How nearest-neighbour search with HNSW and IVF indexes works at scale, and why a vector database is architecturally different from a keyword search index even though both answer 'find me relevant documents'.
From keywords to meaning
A traditional search index like Elasticsearch or a PostgreSQL full-text index answers the question "which documents contain these words" using an inverted index: for every term, a list of documents that contain it, with positions and frequencies, scored typically with BM25, a refinement of TF-IDF that weighs rare terms more heavily and discounts diminishing returns from repeated terms. This is fast and precise for lexical matching but structurally blind to meaning: a search for "cardiac arrest" will not retrieve a document that only says "heart stopped" unless the two phrases happen to share indexed terms, because the inverted index has no representation of semantic similarity at all, only term co-occurrence.
An embedding model solves a different problem: it maps a piece of text (or image, or audio) into a dense vector of typically 384 to 3072 real numbers, trained so that inputs with similar meaning land close together in that vector space under a distance metric such as cosine similarity or Euclidean distance. "Cardiac arrest" and "heart stopped" end up near each other in embedding space even though they share zero words, because the model was trained (usually via contrastive learning on pairs of semantically related and unrelated text) to organise the space by meaning rather than vocabulary. A vector database exists to store millions or billions of these vectors and answer, for any query vector, "which stored vectors are closest to this one" fast enough to serve a user-facing request, which is the retrieval half of retrieval-augmented generation: embed the user's question, find the nearest stored document chunks, and hand those chunks to a language model as context so it can answer grounded in retrieved facts rather than only its training-time knowledge.
Why brute-force nearest neighbour does not scale
The naive way to find the k nearest vectors to a query is to compute the distance from the query to every stored vector and sort — exact k-nearest-neighbour search. This is embarrassingly correct and embarrassingly slow: at a million 768-dimensional vectors, a single query means a million dot products, and at production query volumes (thousands of queries per second across a RAG-backed application) this becomes the dominant cost of the whole system. The problem compounds with what is sometimes called the curse of dimensionality: in high-dimensional spaces, the ratio between the nearest and farthest neighbour's distance shrinks, meaning many of the pruning tricks that work brilliantly for 2D or 3D spatial data (like k-d trees, which partition space along axes) degrade toward brute-force performance once dimensionality gets into the hundreds, because axis-aligned partitioning stops being informative when every point is roughly equidistant from every other point along most axes.
This is why production vector databases use approximate nearest neighbour (ANN) search: algorithms that sacrifice a small, tunable amount of recall (say, finding 98 of the true 100 nearest neighbours instead of all 100) in exchange for orders-of-magnitude speedups. The two dominant families in production systems today are IVF (inverted file index) and HNSW (hierarchical navigable small world graphs), and they represent genuinely different strategies for the same trade-off.
IVF and HNSW: two strategies for approximate search
IVF works by clustering: during index build, the vector space is partitioned into a fixed number of clusters (via k-means), each with a centroid. At query time, the query vector is compared against the cluster centroids first (cheap, because there are far fewer centroids than data points), the nearest few clusters are selected, and the exhaustive distance computation happens only within those selected clusters rather than across the whole dataset. The key tuning parameter is nprobe — how many clusters to search — which directly trades recall for speed: nprobe=1 is very fast and may miss neighbours that happen to sit near a cluster boundary but got assigned to the wrong cluster; a higher nprobe recovers more recall at the cost of more distance computations. IVF is often paired with product quantisation (PQ), which compresses each vector into a short code by splitting it into sub-vectors and replacing each with the nearest of a small learned codebook of centroids, dramatically cutting memory footprint at some cost to distance-computation precision — this combination (IVF-PQ) is what lets FAISS-style indexes hold billions of vectors in memory that would otherwise not fit.
HNSW takes a completely different approach, building a multi-layer graph where each vector is a node connected to its approximate nearest neighbours. The top layer is sparse with long-range connections spanning large regions of the space; each layer below is progressively denser with more local, shorter-range connections, down to the bottom layer which contains every vector. A query starts at an entry point in the sparse top layer, greedily walks toward the query vector (moving to whichever neighbour is closer, like a skip-list for vector space), and drops down a layer once it can't improve further, repeating until it reaches the bottom layer for a fine-grained local search. This gives HNSW excellent recall-versus-latency characteristics, generally better than IVF at the same speed for many workloads, at the cost of higher memory usage (graph edges are expensive to store) and slower index construction, since inserting a new vector requires finding and wiring its approximate neighbours at every layer it participates in. The practical choice between them usually comes down to whether the workload is closer to "static, huge, memory-constrained" (favouring IVF-PQ) or "needs the best recall/latency and can afford more RAM" (favouring HNSW), which is why most managed vector databases (Pinecone, Weaviate, Qdrant, Milvus) default to HNSW but offer IVF-style options for very large collections.
Chunking, hybrid search and the parts that actually determine RAG quality
The index algorithm gets the attention, but in practice the quality of a RAG system is usually bottlenecked elsewhere: how the source documents are chunked before embedding. Embed a whole 20-page PDF as one vector and you get a blurry average representation that matches nothing precisely; chunk too aggressively into single sentences and you lose the surrounding context a retrieved snippet needs to be useful to the downstream language model. Most production systems use overlapping chunks of roughly 200 to 800 tokens, sized to the embedding model's effective context window and often aligned to semantic boundaries (paragraphs, sections) rather than fixed character counts, with a sliding overlap so that a fact spanning a chunk boundary is not lost entirely to one side.
The second underrated factor is that pure vector similarity is not always what a query needs: a user searching for an exact product code or error message wants lexical precision, which is exactly what embeddings are comparatively weak at, since two structurally different strings can still embed close together and an exact string that appears rarely in training data may not get a sharply distinctive vector. This is why most serious RAG deployments use hybrid search — running BM25 keyword search and vector similarity search in parallel and merging results with a reranking step (often a smaller cross-encoder model applied only to the top candidates from each, since cross-encoders are far more accurate than embedding similarity but too slow to run against every document in the collection). The vector database, in other words, is necessary infrastructure for semantic retrieval but rarely sufficient on its own for retrieval quality; the surrounding pipeline of chunking strategy, hybrid retrieval and reranking typically matters as much as the choice of ANN algorithm underneath it.
Frequently Asked Questions
Why can't a normal SQL database just store embeddings in a column and search them?
It can store them, and extensions like pgvector let PostgreSQL do exactly this with IVF or HNSW indexes built in. What a purpose-built vector database typically adds is horizontal scaling for very large collections, more mature ANN tuning options, and native support for metadata filtering combined efficiently with vector search, which becomes important once a collection reaches tens of millions of vectors.
What does 'recall' mean in the context of approximate nearest neighbour search?
Recall here measures what fraction of the true k nearest neighbours an approximate search actually returns, compared to an exact brute-force search. A recall of 0.95 means the ANN index finds 95 percent of the genuinely closest vectors on average, trading the missing 5 percent for substantially lower query latency.
Is a bigger embedding model always better for RAG?
Not necessarily. Larger embedding models often produce marginally better semantic representations but increase both storage cost (more dimensions per vector) and query latency, and the retrieval quality gain frequently matters less than chunking strategy and hybrid search design for overall RAG performance.
What is the difference between cosine similarity and Euclidean distance for embeddings?
Cosine similarity measures the angle between two vectors regardless of their magnitude, which suits embeddings where direction encodes meaning more reliably than length. Euclidean distance measures straight-line distance and is sensitive to vector magnitude; many embedding models are trained specifically to be compared with cosine similarity, and using the wrong metric can degrade retrieval quality.