Word Embeddings and the Semantic Vector Space

Why words need coordinates

A computer cannot compute with the string "honeybee" any more than it can compute with a smell. Before any statistical or neural model can work with text, every word has to become a number, or more precisely a list of numbers — a vector. The earliest attempts at this, such as one-hot encoding, gave each word in a vocabulary its own dedicated slot in a vector as long as the vocabulary itself: "cat" might be a 1 followed by fifty thousand zeros, "dog" a zero followed by a 1 and forty-nine thousand nine hundred ninety-eight more zeros. This representation is easy to build but useless for meaning — every word is exactly as different from every other word, whether it is "dog" versus "puppy" or "dog" versus "thermodynamics". There is no sense in which the vector for "dog" is closer to "puppy" than to "spreadsheet".

Word embeddings solve this by learning a much shorter vector for each word — typically 100 to 300 numbers rather than tens of thousands — where the position of the vector in space is not arbitrary but reflects how the word is actually used in language. Two words that tend to appear in similar contexts end up with vectors that point in similar directions. This single idea, sometimes called the distributional hypothesis ("a word is characterized by the company it keeps", a phrase coined by linguist John Firth in 1957), is the foundation nearly all of modern natural language processing is built on.

Learning vectors by predicting context: Word2Vec

The technique that popularized dense word embeddings at scale was Word2Vec, introduced by a team at Google in 2013. Word2Vec does not try to define what a word means; it trains a small neural network to solve a much simpler proxy task, and the useful embeddings fall out as a side effect. There are two variants. In the skip-gram version, the network is shown a target word and asked to predict the words that surround it in a sentence — given "honeybee", predict that words like "hive", "pollen" and "nectar" are likely to be nearby. In the continuous bag-of-words (CBOW) version, the task runs in reverse: given the surrounding words, predict the missing target word in the middle.

Neither task is interesting in itself — nobody needs a model that predicts context words. What matters is that to get good at this prediction task, the network is forced to compress each word into a vector that captures its typical usage. Words that show up in the same kinds of sentences, like "queen" and "monarch", or "nurse bee" and "worker", end up nearby in the resulting vector space purely because the training process pushed them there for statistical reasons.

GloVe (Global Vectors), developed at Stanford shortly afterward, takes a related but distinct approach: rather than predicting local context word by word, it directly factorizes a matrix of how often word pairs co-occur across an entire corpus, which lets it capture broader statistical patterns more directly. In practice Word2Vec and GloVe vectors behave similarly and are often interchangeable in applications.

Geometry that means something: analogies and distance

The remarkable discovery that made word embeddings famous outside of NLP research circles was that the vector space they learn has genuine algebraic structure. If you take the vector for "king", subtract the vector for "man", and add the vector for "woman", the resulting point in space lands very close to the vector for "queen". The direction from "man" to "woman" appears to encode something like "gender", and that same direction, applied elsewhere, encodes the analogous relationship. Similar patterns show up for verb tense ("walk" to "walked" mirrors "swim" to "swam"), for country-capital pairs ("France" to "Paris" mirrors "Japan" to "Tokyo"), and for pluralization.

This does not mean the model understands gender, geography or grammar the way a person does. It means that these relationships are consistently reflected in how words are used across millions of sentences, and the training process is sensitive enough to encode that consistency as a roughly linear direction in the vector space. It is an emergent property of the statistics of language, not a hand-built rule.

Because meaning is now geometry, similarity becomes a measurable quantity. The standard measure is cosine similarity, which looks at the angle between two vectors rather than their raw distance (so it is not thrown off by words that simply appear more often and therefore have longer vectors). A cosine similarity near 1 means two words point in almost the same direction and are used in very similar contexts; a value near 0 means they are largely unrelated; a value near -1, which is rare for word vectors in practice, would mean opposite usage patterns. This is exactly the kind of relationship a 3D visualization can make intuitive: plotting word vectors (after reducing 300 dimensions down to three with a technique like PCA or t-SNE) lets you see clusters of related words — fruits, professions, countries — forming visibly separate neighbourhoods in space, and lets you watch an analogy like king − man + woman literally trace a path that lands near queen.

The limits of one vector per word

Classic embeddings like Word2Vec and GloVe are static: the word "bank" gets exactly one vector, which has to somehow average together its meaning as a financial institution and its meaning as the edge of a river, because the model has no way to represent context. This is a real limitation. FastText, an extension developed at Facebook, improved on this at the subword level by learning vectors for character n-grams as well as whole words, so it could construct a reasonable vector for a word it had never seen during training (useful for typos, rare words, and morphologically rich languages) — but it still assigns one fixed vector per surface form, so it does not solve the ambiguity problem.

The real fix came with contextual embeddings. Models like ELMo (2018) used bidirectional LSTMs to produce a different vector for "bank" depending on the sentence it appeared in, and BERT and GPT, both built on the transformer architecture introduced in 2017, took this further by generating embeddings through many layers of self-attention that let every word's representation be influenced by every other word in the sentence simultaneously. In a contextual model, "bank" in "I deposited a cheque at the bank" and "bank" in "we sat on the river bank" produce genuinely different vectors, computed on the fly, rather than sharing a single static one. Static embeddings remain useful as a fast, lightweight starting point or baseline; contextual embeddings are what powers state-of-the-art language understanding today.

What embeddings are used for

Word embeddings (static or contextual) are rarely the end product of an NLP system; they are the input layer that everything else builds on. Downstream uses include: measuring document similarity by averaging or otherwise combining the embeddings of the words in each document and comparing the results; powering search and recommendation systems that need to match a query to relevant results even when the exact words differ ("laptop" retrieving results about "notebook computers"); providing the numeric input to text classifiers for sentiment analysis, topic labelling or spam detection; and initializing the input layer of larger neural networks for translation, question answering or summarization, so the network does not need to relearn basic word relationships from scratch.

Word embeddings are also a rich source of insight into bias in training data, because the geometry reflects the corpus a model was trained on, warts and all. Analogy experiments run on embeddings trained on news text or web crawl data have repeatedly surfaced gender and racial biases baked into common usage patterns (for example directions that associate certain professions more strongly with one gender than another), which is why bias auditing of embeddings became an active and necessary area of NLP research once these vector spaces started driving real-world decisions.