Every embedding here is a unit vector on a circle (in a real model it lives on a 512- or 2048-dimensional hypersphere — the dimensionality is arbitrary to the theory; L2-normalizing after every update is the standard trick used by SimCLR and MoCo, and a 2D circle preserves the exact same cosine-similarity and tangent-projection math as the 3D sphere, just with one fewer axis to look at). Each of the 6 coloured classes has a query embedding q (moved by "training") and a key embedding k, produced by a slowly-updated momentum encoder.
Each step, one class is sampled as the anchor. Its positive key k⁺ is its own momentum-encoder embedding; its negatives are every key currently sitting in the FIFO queue (this decouples negative count from batch size — MoCo's key idea over SimCLR, which needs a huge batch to get enough negatives):
InfoNCE: L = -log( exp(sim(q,k⁺)/τ) / Σᵢ exp(sim(q,kᵢ)/τ) )
sim(a,b) = a·b (cosine similarity of unit vectors)
The query is nudged along the InfoNCE gradient projected onto the circle's tangent line — toward k⁺ in proportion to (1 − p⁺), and away from every negative kᵢ in proportion to its softmax weight pᵢ — then renormalized back onto the circle:
∇q ∝ (1-p⁺)·k⁺ − Σᵢ pᵢ·kᵢ
q ← normalize(q + η·tangential(∇q))
The momentum (key) encoder never trains directly — it drifts toward the query encoder by a small fraction each step, which is what makes it a slow, consistent source of negatives instead of a moving target:
k ← normalize(m·k + (1-m)·q)
- Temperature τ — lower values sharpen the softmax, punishing near-miss negatives harder and pulling clusters tighter.
- Momentum m — closer to 1 means the key encoder barely moves each step, giving very consistent (but slower-adapting) negatives.
- Queue size K — more negatives per step, at the cost of holding more stale (older) embeddings in memory.
- Step size η — how far the query jumps along the gradient each training step.
- Auto-train speed — how many training steps run per second while auto-training is on.
Drag inside the main circle to rotate the view. Watch the six classes separate into tight, well-spread clusters on the circle as training proceeds — exactly the geometric effect contrastive pretraining (SimCLR, MoCo, and CLIP's image-text version) has on real embedding spaces before any labelled fine-tuning ever happens. The loss curve (top right) and the negative-similarity histogram (bottom right) track that separation numerically.