A large language model is trained once, on a fixed snapshot of text, and then frozen. It has no idea what your refund policy says, what changed in your API last week, or what is in the PDF a customer just uploaded. Retrieval-augmented generation (RAG) is the standard pattern for closing that gap: instead of asking the model to answer from memory alone, you first retrieve relevant documents and hand them to the model as grounding context.
The pipeline, step by step
A query comes in. It gets converted into a vector representation (an embedding) that captures its meaning. That vector is compared against embeddings of every document in a corpus, using a similarity measure such as cosine similarity. The top-k most similar documents are retrieved and inserted into the prompt, ahead of the actual question, so the model answers using text that is actually in front of it rather than recalled from training.
Why retrieval quality is everything
If the retrieved documents are irrelevant, the model has two options: hallucinate an answer or admit it does not know. Groundedness, how directly the retrieved context actually supports the answer, is the single biggest lever in a RAG system. A bigger corpus with weak retrieval performs worse than a small corpus with strong retrieval.
Choosing k
Retrieving too few documents (a small k) risks missing the one that actually answers the question. Retrieving too many risks diluting the prompt with irrelevant text and pushing the truly relevant passage out of the model's effective attention. Most production systems tune k empirically, often landing between 2 and 5.
Try it yourself
The LLM Integration Lab lets you pick a query, adjust k, and watch a simplified keyword-overlap similarity score decide which of eight mock documents get retrieved, with a live "estimated groundedness" readout.
🧪 Try it yourself: the LLM Integration Lab simulation lets you experiment with everything described above directly in your browser.