Contrastive Image-Text Embedding: How CLIP Learns to Match Pictures and Words

A Shared Space for Pictures and Words

At the heart of CLIP is a simple but powerful idea: convert both images and text into vectors, or embeddings, that live in the same high-dimensional space. Two separate neural networks do the conversion, one for images and one for text, but they are trained so that the resulting vectors are directly comparable. When an image and its caption describe the same thing, their embeddings should point in nearly the same direction, so a simple similarity score, like the cosine of the angle between them, tells the model how well a picture and a phrase match. Unrelated pairs, like a photo of a dog matched with the caption “a plate of pasta”, should end up far apart in this space.

Training by Contrast: InfoNCE in Action

CLIP is not trained on one image-caption pair at a time. Instead, it processes a whole batch of pairs at once, say 256 images and their 256 true captions, and encodes every image and every caption into vectors. It then builds a full grid of similarity scores between every image and every caption in the batch, and trains the networks so that each image's true caption scores higher than all the other captions in that batch, while each caption's true image scores higher than all the other images. This is the InfoNCE contrastive loss: it rewards correct pairs and penalises every mismatched combination simultaneously, which is far more informative per training step than simply saying “this pair matches” or “this pair does not”.

Why Noisy Internet Data Was the Real Breakthrough

Traditional image classifiers were trained on carefully hand-labelled datasets with a fixed, limited set of categories, like 1,000 ImageNet classes. CLIP instead trained on around 400 million image-caption pairs scraped from the public internet, where captions are messy, informal, and cover an enormous range of concepts, styles, and phrasing. Because the model learns to connect images with natural language rather than a rigid label list, it picks up a much richer notion of visual concepts, and can generalise to categories it never saw an explicit example of during training. This gives CLIP strong zero-shot performance: at test time you can simply describe a new category in words, and the model compares an image against those text descriptions with no additional training.

From Research Idea to Everyday AI Tool

This kind of joint embedding space now underpins zero-shot image classification, text-to-image and image-to-text search engines, content moderation, and it famously provides the “steering” signal that guides text-to-image generators like DALL-E and Stable Diffusion toward prompts. However, the approach inherits whatever biases and stereotypes are present in its internet-scraped training data, its performance can be surprisingly sensitive to how a prompt is phrased, and matching a caption to an image is not the same as genuinely understanding either one. Researchers and engineers who deploy these embeddings need to actively test for these blind spots rather than assume the model reasons the way a human would.

Frequently asked questions

Try it live

Everything above runs in your browser — open Contrastive Image-Text Embedding: How CLIP Learns to Match Pictures and Words and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.