Momentum Contrast (MoCo): The Negative-Pair Queue

Training Loop

Hyperparameters

Temperature τ 0.20

Momentum m 0.990

Queue size K 64

Step size η 0.25

Live Readouts

InfoNCE loss

—

Training steps

Positive cos-sim

—

Mean negative cos-sim

—

How it works

Every embedding here is a unit vector on a 3D sphere (in a real model it lives on a 512- or 2048-dimensional hypersphere; L2-normalizing after every update is the standard trick used by SimCLR and MoCo). Each of the 6 coloured classes has a query embedding q (moved by "training") and a key embedding k, produced by a slowly-updated momentum encoder.

Each step, one class is sampled as the anchor. Its positive key k⁺ is its own momentum-encoder embedding; its negatives are every key currently sitting in the FIFO queue (this decouples negative count from batch size — MoCo's key idea over SimCLR, which needs a huge batch to get enough negatives):

InfoNCE:  L = -log( exp(sim(q,k⁺)/τ) / Σᵢ exp(sim(q,kᵢ)/τ) )
sim(a,b) = a·b               (cosine similarity of unit vectors)

The query is nudged along the InfoNCE gradient projected onto the sphere's tangent plane — toward k⁺ in proportion to (1 − p⁺), and away from every negative kᵢ in proportion to its softmax weight pᵢ — then renormalized back onto the sphere:

∇q ∝ (1-p⁺)·k⁺ − Σᵢ pᵢ·kᵢ
q  ← normalize(q + η·tangential(∇q))

The momentum (key) encoder never trains directly — it drifts toward the query encoder by a small fraction each step, which is what makes it a slow, consistent source of negatives instead of a moving target:

k ← normalize(m·k + (1-m)·q)

Temperature τ — lower values sharpen the softmax, punishing near-miss negatives harder and pulling clusters tighter.
Momentum m — closer to 1 means the key encoder barely moves each step, giving very consistent (but slower-adapting) negatives.
Queue size K — more negatives per step, at the cost of holding more stale (older) embeddings in memory.
Step size η — how far the query jumps along the gradient each training step.

Watch the six classes separate into tight, well-spread clusters on the sphere as training proceeds — exactly the geometric effect contrastive pretraining (SimCLR, MoCo, and CLIP's image-text version) has on real embedding spaces before any labelled fine-tuning ever happens.

Query embedding q Key embedding k (momentum) Queue negatives (fade = older)

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.