Natural Language Processing: From Text to Understanding

Foundations

NLP: intersection of linguistics, computer science, and AI — enabling machines to understand and generate human language. Text preprocessing: tokenization (word, subword — BPE, WordPiece, SentencePiece), lowercasing, stemming (Porter), lemmatization (WordNet), stop word removal. Bag of Words (BoW): document as word frequency vector, ignoring order. TF-IDF: term frequency × inverse document frequency — measures word importance in document vs. corpus. N-grams: contiguous sequences of n tokens, capturing local context. Named Entity Recognition (NER): identifying entities (persons, organizations, locations) — sequence labeling with BIO tags. Part-of-speech tagging: grammatical role assignment (noun, verb, adjective).

Word Embeddings

Word2Vec (Mikolov et al., 2013): dense vector representations capturing semantic relationships. Architectures: CBOW (predict word from context), Skip-gram (predict context from word). Famous example: king - man + woman ≈ queen (vector arithmetic captures analogies). GloVe (Pennington et al., 2014): Global Vectors from co-occurrence matrix factorization. FastText (Bojanowski et al., 2017): subword embeddings — handles morphology and OOV words. ELMo (Peters et al., 2018): contextualized embeddings from bidirectional LSTM — different representations for polysemous words. Evaluation: intrinsic (word similarity, analogies) and extrinsic (downstream task performance).

Transformers

Transformer (Vaswani et al., 2017): "Attention Is All You Need" — replaced RNNs for sequence modeling. Self-attention: each token attends to all other tokens, computing weighted sum. Multi-head attention: parallel attention with different projections (typically 8-16 heads). Positional encoding: sinusoidal or learned — injects sequence order information. Architecture: encoder (bidirectional) + decoder (autoregressive), residual connections, layer normalization. Computational complexity: O(n²d) for sequence length n and dimension d. BERT (Devlin et al., 2019): bidirectional encoder, pre-trained with MLM (Masked Language Model) and NSP. GPT series (OpenAI): decoder-only, autoregressive pre-training on next-token prediction. T5 (Google): encoder-decoder, text-to-text framework for all NLP tasks.

Large Language Models

Scaling laws (Kaplan et al., 2020): loss decreases as power law with model size, data, and compute. GPT-3 (175B parameters, 2020): in-context learning, few-shot capabilities. GPT-4 (2023): multimodal (text + vision), improved reasoning. Chinchilla (Hoffmann et al., 2022): optimal compute allocation — more data, smaller model. LLaMA (Meta): open-weight models, competitive with larger proprietary models. Fine-tuning: supervised fine-tuning (SFT) + RLHF (Reinforcement Learning from Human Feedback). Prompt engineering: zero-shot, few-shot, chain-of-thought prompting. RAG (Retrieval-Augmented Generation): combining LLMs with external knowledge retrieval. Limitations: hallucinations, context window limits, lack of true reasoning, training data cutoff.

Applications

Machine translation: neural MT (encoder-decoder with attention), Google Translate serves billions daily. Sentiment analysis: opinion mining for product reviews, social media monitoring. Question answering: extractive (SQuAD benchmark) and generative (open-domain QA). Text summarization: extractive (sentence selection) and abstractive (generation of new text). Chatbots and assistants: task-oriented (booking, customer service) and open-domain conversation. Code generation: Codex, GitHub Copilot, StarCoder — LLMs trained on code. Information extraction: relation extraction, event detection, knowledge graph construction. Biomedical NLP: PubMedBERT, clinical NER, drug interaction detection. Multilingual NLP: mBERT, XLM-RoBERTa — cross-lingual transfer learning.

Frequently Asked Questions

What is NLP?

Natural Language Processing is a field of AI that enables computers to understand, interpret, and generate human language, combining techniques from linguistics, computer science, and machine learning.

What is a transformer?

A transformer is a neural network architecture based on self-attention mechanisms that processes all tokens in parallel, enabling efficient training on large datasets and forming the basis of modern LLMs.

What is BERT?

BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model that reads text bidirectionally, achieving state-of-the-art results on many NLP tasks through fine-tuning.

What are word embeddings?

Word embeddings are dense vector representations of words in a continuous space where semantically similar words are closer together, capturing relationships like synonyms and analogies.

What is RAG?

Retrieval-Augmented Generation combines large language models with external knowledge retrieval, reducing hallucinations and enabling access to up-to-date or domain-specific information.

Try it live

Everything above runs in your browser — open Dimensionality Reduction: PCA, t-SNE & UMAP and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.