t-SNE & UMAP

Nonlinear dimensionality reduction that preserves local neighborhoods — the go-to tools for visualizing high-dimensional embeddings in 2D or 3D.

1. Why Not Just PCA?

PCA captures directions of maximum linear variance. When data lies on a curved manifold (images, word embeddings, single-cell RNA), PCA squashes structure together. t-SNE and UMAP are nonlinear methods that preserve local neighborhoods: points close in high-D stay close in 2D.

2. t-SNE — How It Works

In high-D: convert pairwise distances into conditional probabilities p(j|i) via a Gaussian centered at each point.
In 2D: place points and compute analogous probabilities q(j|i) using a Student's-t distribution (heavy tails to avoid crowding).
Minimize the KL divergence between P and Q with gradient descent.

min KL(P ‖ Q) = Σᵢⱼ pᵢⱼ · log(pᵢⱼ / qᵢⱼ)

Key hyperparameter — perplexity

Roughly the effective number of neighbors each point considers. Typical range 5–50. Small → focuses on very local structure; large → smoother global structure.

3. UMAP — Faster, Often Better Global Structure

UMAP builds a fuzzy weighted neighborhood graph in high-D and optimizes a low-D layout whose graph matches it (cross-entropy loss). It's faster than t-SNE, scales to millions of points, and tends to preserve more global structure.

n_neighbors — local vs global balance (default 15). Higher → more global.
min_dist — how tightly points pack together (default 0.1).
metric — distance function (Euclidean, cosine, Manhattan, …).

4. Python Implementation

from sklearn.manifold import TSNE
from sklearn.decomposition import PCA
import umap   # pip install umap-learn

# Pre-reduce with PCA before t-SNE for speed & noise reduction
X_pca = PCA(n_components=50, random_state=0).fit_transform(X)

X_tsne = TSNE(
    n_components=2, perplexity=30, learning_rate="auto",
    init="pca", random_state=0,
).fit_transform(X_pca)

X_umap = umap.UMAP(
    n_neighbors=15, min_dist=0.1, metric="euclidean", random_state=0,
).fit_transform(X)

5. t-SNE vs UMAP vs PCA

PCA: linear, fast, deterministic, axes interpretable. Good first step.
t-SNE: nonlinear, great local structure, slow (O(N²) → Barnes-Hut O(N log N)), distorts global distances.
UMAP: nonlinear, faster, scales better, better global structure, supports new-point transform.

6. Common Pitfalls

Cluster sizes and inter-cluster distances are NOT meaningful in t-SNE — only local neighborhoods.
Different perplexity/seed → very different-looking maps. Always show a few.
Don't use t-SNE/UMAP embeddings as features for downstream models — use PCA or autoencoders for that.
Always standardize / L2-normalize features (or use cosine metric) before fitting.

Mental model

Use PCA for compression and as a preprocessing step. Use t-SNEfor beautiful local visualizations on smaller datasets. Use UMAP as the general-purpose default for visualizing modern embeddings at scale.