🎨 t-SNE Visualization

Understanding t-SNE

t-Distributed Stochastic Neighbor Embedding (t-SNE) is a powerful technique for visualizing high-dimensional data. Unlike PCA, t-SNE is non-linear and excels at preserving local structure, making clusters visually apparent.

How t-SNE Works

The algorithm has two main phases:

• Phase 1 - High-Dimensional Similarities:
  • Compute pairwise similarities in original space
  • Use Gaussian kernel: p_ij ∝ exp(-||x_i - x_j||² / 2σ²)
  • Convert to conditional probabilities
  • Perplexity parameter controls σ
• Phase 2 - Low-Dimensional Embedding:
  • Random initialization in 2D/3D
  • Compute similarities using t-distribution: q_ij ∝ (1 + ||y_i - y_j||²)^(-1)
  • Minimize KL divergence: KL(P||Q) = Σ p_ij log(p_ij / q_ij)
  • Gradient descent optimization

Why t-Distribution?

t-SNE uses Student's t-distribution (instead of Gaussian) in low dimensions:

• Crowding Problem: High-dim space has more "room" than 2D
• Heavy Tails: t-distribution has heavier tails than Gaussian
• Benefit: Dissimilar points can be far apart without penalty
• Result: Better separation between clusters

Perplexity Parameter

Perplexity is the most important t-SNE parameter:

• Interpretation: Rough measure of effective number of neighbors
• Effect:
  • Low perplexity (5-10): Emphasizes local structure, many small clusters
  • Medium perplexity (20-50): Balanced, typical choice
  • High perplexity (>50): Emphasizes global structure
• Guideline: Try values between 5 and 50, often 30 works well
• Dataset Size: Larger datasets can use higher perplexity

t-SNE vs PCA

• Linearity:
  • PCA: Linear projection
  • t-SNE: Non-linear manifold learning
• Global vs Local:
  • PCA: Preserves global structure (distances)
  • t-SNE: Preserves local structure (neighborhoods)
• Speed:
  • PCA: Fast (O(np²))
  • t-SNE: Slow (O(n² log n) best case)
• Deterministic:
  • PCA: Always same result
  • t-SNE: Stochastic (different runs differ)
• Interpretability:
  • PCA: Axes are principal components
  • t-SNE: Axes have no meaning

Best Practices

• Preprocessing:
  • Apply PCA first to reduce to ~50 dimensions
  • Speeds up t-SNE significantly
  • Remove noise dimensions
• Try Multiple Runs:
  • t-SNE is stochastic - results vary
  • Run 3-5 times with different random seeds
  • Look for consistent patterns
• Tune Perplexity:
  • Try 5, 30, 50 to see different structures
  • No single "correct" value
• Enough Iterations:
  • At least 1000 iterations
  • Monitor KL divergence convergence
  • If still decreasing, run more
• Don't Over-Interpret:
  • Cluster sizes not meaningful
  • Distances between clusters not meaningful
  • Only local neighborhoods reliable

Common Pitfalls

• Interpreting Distances:
  • Don't compare distances between separated clusters
  • Global distances not preserved
• Cluster Sizes:
  • Size/density not meaningful
  • Don't conclude "more important" from size
• Single Run:
  • One run may be misleading
  • Always try multiple runs
• Too Few Iterations:
  • Early iterations look messy
  • Need convergence for meaningful results

Applications

• Data Exploration: Visualize high-dim datasets (images, embeddings)
• Cluster Validation: See if clusters are well-separated
• Anomaly Detection: Outliers appear as isolated points
• Feature Learning: Visualize learned representations (neural network embeddings)
• Single-Cell Genomics: Visualize cell types
• NLP: Visualize word/document embeddings
• Computer Vision: Visualize image features

Alternatives to t-SNE

• UMAP (Uniform Manifold Approximation and Projection):
  • Faster than t-SNE
  • Better preserves global structure
  • More scalable
  • Increasingly popular alternative
• PacMAP:
  • Pairwise Controlled Manifold Approximation
  • Balances local and global structure
  • Faster than t-SNE
• LargeVis:
  • Optimized for large datasets
  • Millions of points

Optimizations

• Barnes-Hut Approximation:
  • Reduces complexity from O(n²) to O(n log n)
  • Uses tree structure for far points
  • Enables datasets with 10,000s points
• Fast Fourier Transform (FFT):
  • Further speed improvements
  • FIt-SNE implementation

Implementation Tips

• Use scikit-learn's TSNE or openTSNE (faster)
• Apply PCA to ~50 dims first
• Start with perplexity=30, learning_rate=200
• Run for 1000+ iterations
• Use init='pca' for better initialization
• For large datasets (>10K), use Barnes-Hut or UMAP
• Color points by known labels for validation

Experiment with the Demo

Use the interactive tool above to:

• See t-SNE iteratively optimize embedding
• Adjust perplexity and observe effects
• Watch clusters emerge during optimization
• Compare different datasets
• Understand non-linear dimensionality reduction

t-SNE is invaluable for exploring and understanding high-dimensional data. It reveals structure that linear methods like PCA miss!