This is the vector-space model that classical search engines and text-mining pipelines use to rank documents against a query — the same math behind TF-IDF-based information retrieval before neural embeddings existed, and still used today for fast keyword search and as a baseline.
tf(t,d) = count(t in d) / |d|
idf(t) = ln(N / df(t))
tfidf(t,d) = tf(t,d) · idf(t)
cos(q,d) = (q·d) / (‖q‖ ‖d‖)
- Tokenizing — your query is lowercased and split on non-letters; words outside the fixed 10-document corpus vocabulary are dropped and shown as unrecognized.
- TF-IDF vs TF only — TF-IDF down-weights terms that appear in many documents (idf) so common words don't dominate ranking; TF-only weights every matching term equally, which is why rankings shift when you toggle it.
- Cosine similarity — ranking is computed exactly in the full vocabulary-dimensional space (~51 terms), never in the 2D view.
- 2D positions — each vocabulary term is mapped to a fixed unit direction on a circle via the golden angle (a deterministic random projection, Johnson–Lindenstrauss style — the 2D analogue of a spherical Fibonacci lattice). Document and query points are the weighted sum of their terms' directions, so nearby points are usually similar — but the scatter plot is an approximation; the ranked list on the left and the bar chart below are the ground truth.
- Inverted index — for each term, the list of document ids containing it; this is what a real search engine stores instead of scanning every document per query.