Self-supervised contrastive learning on a circle (2D)

Two views of the same example are pulled together, everything else is pushed apart. Watch classes appear that no label ever named.

Switch to 3D

Presets

Playback

Loss

Augmentations and data

Layers

Model check

The science

Labels made from the data itself

Contrastive learning needs no class labels. Each training example is turned into two random views (here: jitter on the two signal features, plus fresh background noise on four nuisance features). The encoder maps every view to a point z = Wφ(x) / |Wφ(x)| on the unit circle (2D page) or sphere (3D page). Views of the same instance are positives; all other views in the batch are negatives. The four colours are only used to draw the picture and to score a probe; the training never sees them.

The InfoNCE (NT-Xent) loss

For a batch of B instances there are 2B views. For view i with positive p(i) and cosine similarity s_ik = z_i . z_k:

L = (1/2B) Σ_i [ -s_ip/τ + ln Σ_{k≠i} exp(s_ik/τ) ]

The temperature τ sets how sharply hard negatives are punished. A small τ spreads the embedding out (strong uniformity, but also pushes apart same-class instances); a large τ tolerates clusters. The analytic gradient (through the softmax and through the length normalisation) is trained by SGD with momentum 0.9. Check 1 compares every weight's gradient with a central finite difference.

Collapse and the log(2B-1) ceiling

If the encoder maps everything to one point, every similarity is 1 and the loss equals exactly ln(2B-1); training starts near this value and must beat it. Check 2 verifies the identity and that perfectly spread, aligned pairs score far below it.

Alignment and uniformity

Wang and Isola (2020) split the objective into alignment E|z - z+|^2 (positives close) and uniformity ln E exp(-t|z_i - z_j|^2) with t = 2 (points spread over the sphere). For a perfectly uniform circle the expectation is e^-4 I0(4), for a uniform sphere (1 - e^-8)/8; check 3 compares the engine's uniformity of an evenly spread point set with these closed forms.

Probe and shortcuts

The probe is a nearest-class-mean classifier on the frozen embedding: class means come from 400 labelled examples and accuracy is measured on 240 others. Check 4 runs a fixed-seed, 150-step training and requires the loss to fall below the collapse value and the probe to beat chance (25 %). The shortcut slider adds an instance-specific signal that is identical in both views: the encoder can match positives by copying it and learn less about the real classes, which is a known failure mode of contrastive learning.

Honest simplifications

A linear encoder on nine hand-made features (two signals, four nuisances, three quadratic terms), four classes, no projection head and no deep network. Real systems (SimCLR, MoCo, CLIP) use deep encoders, millions of images and far larger batches, but the loss, its temperature and the alignment/uniformity picture are the same. Numbers are illustrative.

InfoNCE loss against the collapse value ln(2B-1)
Alignment and uniformity (Wang-Isola) during training
Class-mean probe accuracy (labels used only to score)
Cosine similarity: positive pairs against all others
Made with 🟠 Blender · Three.js