🔤 Word Embeddings Explorer

Interactive NLP Vector Space Visualization

2D Word Embedding Space

Add Words

Vector Arithmetic

Similarity Search

Understanding Word Embeddings

Word embeddings are dense vector representations of words that capture semantic meaning. Words with similar meanings have similar vectors, enabling machines to understand language mathematically.

Why Word Embeddings?

Traditional NLP used one-hot encoding: each word is a sparse vector with all zeros except one position. Problems:

  • No notion of similarity (all words equally different)
  • Vocabulary size = vector dimension (huge, sparse)
  • No semantic relationships captured

Embeddings solve this by mapping words to dense, low-dimensional continuous vectors (typically 50-300 dimensions) where semantic similarity = vector proximity.

Word2Vec (2013)

Word2Vec by Mikolov et al. revolutionized NLP with two architectures:

  • CBOW (Continuous Bag of Words):
    • Predict target word from context words
    • Input: surrounding words → Output: center word
    • Faster, better for frequent words
  • Skip-gram:
    • Predict context words from target word
    • Input: center word → Output: surrounding words
    • Better for rare words and small datasets

Key Insight: "You shall know a word by the company it keeps" - words appearing in similar contexts have similar meanings.

GloVe (2014)

GloVe (Global Vectors) by Stanford combines:

  • Global matrix factorization (like LSA)
  • Local context window methods (like Word2Vec)
  • Trains on word co-occurrence statistics
  • Often performs better than Word2Vec
  • Pre-trained on Wikipedia + Gigaword corpus

FastText (2017)

Facebook's FastText improves Word2Vec:

  • Represents words as bag of character n-grams
  • Handles out-of-vocabulary words
  • Better for morphologically rich languages
  • Can generate embeddings for misspellings

The Amazing Vector Arithmetic

Embeddings capture semantic relationships through vector arithmetic:

  • Classic Example:
    • king - man + woman ≈ queen
    • Paris - France + Italy ≈ Rome
    • good - bad + ugly ≈ beautiful
  • Why It Works:
    • Vectors capture relational information
    • Similar relationships = similar vector directions
    • "Male-to-female" is consistent vector direction
    • "Country-to-capital" is consistent direction

Properties of Word Embeddings

  • Semantic Similarity:
    • Cosine similarity measures relatedness
    • Similar words cluster together
    • "dog" close to "cat", "puppy", "pet"
  • Analogies:
    • Linear relationships encode concepts
    • Gender, verb tense, plurals, etc.
  • Dimensionality:
    • Typically 50-300 dimensions
    • Each dimension captures some semantic aspect
    • Not interpretable individually

Training Word Embeddings

  • Data Requirements:
    • Large text corpus (millions-billions of words)
    • Wikipedia, news articles, books, web text
    • More data = better embeddings
  • Preprocessing:
    • Tokenization
    • Lowercase (usually)
    • Remove rare words (or use subword units)
  • Training Parameters:
    • Vector dimension (50-300)
    • Context window size (5-10 words)
    • Negative sampling (Word2Vec)
    • Training epochs (5-15)

Applications

  • Text Classification: Use embeddings as features
  • Information Retrieval: Find similar documents
  • Recommendation Systems: Content-based filtering
  • Machine Translation: Initialize encoder/decoder
  • Named Entity Recognition: Feature representation
  • Sentiment Analysis: Capture nuanced meanings
  • Question Answering: Match questions to answers

Limitations

  • Context-Independent:
    • One vector per word (ignores polysemy)
    • "bank" (river) vs "bank" (financial) same vector
    • Solution: Contextual embeddings (ELMo, BERT)
  • Bias:
    • Embeddings reflect corpus biases
    • Gender stereotypes (doctor-man, nurse-woman)
    • Ethnic biases in sentiment
    • Active research area for debiasing
  • Out-of-Vocabulary:
    • No vector for unseen words
    • FastText partially solves this

Contextual Embeddings (Modern)

Modern models generate context-dependent embeddings:

  • ELMo (2018):
    • Embeddings from Language Models
    • Bi-directional LSTM
    • Different vectors for same word in different contexts
  • BERT (2018):
    • Bidirectional Transformer
    • Pre-trained on masked language modeling
    • State-of-the-art on many NLP tasks
  • GPT Series:
    • Autoregressive transformers
    • Powerful generation capabilities

Evaluation Metrics

  • Intrinsic Evaluation:
    • Word similarity tasks (SimLex-999, WordSim-353)
    • Analogy tasks (Google analogy dataset)
    • Correlation with human judgments
  • Extrinsic Evaluation:
    • Performance on downstream tasks
    • Classification accuracy
    • Translation quality

Visualization Techniques

  • t-SNE: Nonlinear dimensionality reduction for 2D/3D plots
  • PCA: Linear projection to principal components
  • UMAP: Modern alternative to t-SNE, faster and preserves structure

Pre-trained Embeddings

  • Word2Vec: Google News (3B words, 3M vocab)
  • GloVe: Wikipedia + Gigaword (6B tokens)
  • FastText: Common Crawl (600B tokens), 157 languages
  • When to Use:
    • Small dataset: always use pre-trained
    • Large dataset: fine-tune or train from scratch
    • Domain-specific: consider domain-specific embeddings

Implementation Tips

  • Start with pre-trained embeddings (GloVe, FastText)
  • Freeze embeddings initially, fine-tune later if needed
  • Use padding/masking for variable-length sequences
  • Consider subword tokenization (BPE) for better coverage
  • Normalize vectors (unit length) before similarity computation
  • Use approximate nearest neighbor search for efficiency (FAISS)

Multilingual Embeddings

  • Map embeddings from different languages to shared space
  • Enable cross-lingual transfer
  • MUSE (Facebook), LASER, mBERT
  • Applications: zero-shot translation, multilingual classification

Domain-Specific Embeddings

  • BioWordVec: Biomedical text
  • SciBERT: Scientific papers
  • FinBERT: Financial documents
  • ClinicalBERT: Clinical notes
  • Better performance on domain tasks

Experiment with the Explorer

Use the interactive tool above to:

  • Visualize words in 2D embedding space
  • See semantic clusters (animals, countries, emotions)
  • Perform vector arithmetic (king - man + woman = ?)
  • Find similar words based on vector proximity
  • Understand how semantic relationships are captured

Word embeddings are the foundation of modern NLP. Understanding how words are represented as vectors is crucial for building any language AI system!