Labelled data is expensive: someone has to look at each example and assign the correct answer. Unlabelled data, by contrast, is often abundant and nearly free to collect. Self-supervised learning exploits this imbalance by having a model learn useful structure from unlabelled data first, before it ever sees a labelled example, using a pretext task the model can generate correct answers for on its own.
What makes a good pretext task
A pretext task is any problem where the "right answer" can be derived automatically from the data itself, with no human annotation required: predicting a masked word from its surrounding context, recognizing that two augmented views came from the same original image, or discovering which unlabelled points naturally cluster together. Solving these self-generated tasks forces the model to learn a representation that captures real structure in the data.
Why this reduces labelled-data needs
Once pretrained this way, the model already understands much of the underlying structure relevant to the eventual task. Fine-tuning on a small labelled dataset then only has to adjust that existing representation toward the specific target, rather than learning everything from a blank slate, which is why self-supervised models typically need far fewer labelled examples to reach strong accuracy.
Where this shows up in practice
Nearly every large language model is trained this way, using next-word prediction as its pretext task on vast amounts of unlabelled text. Many vision models similarly pretrain by predicting relationships between augmented views of the same image, before any labelled fine-tuning happens at all.
Try it yourself
The Self-Supervised Learning Lab compares a self-supervised-pretrained model against one trained from scratch, letting you adjust how many labelled examples are available and watch the accuracy gap between them.
🧪 Try it yourself: the Self-Supervised Learning Lab simulation lets you experiment with everything described above directly in your browser.