Active learning is a training loop where the model itself decides which unlabelled examples are most worth labelling next, instead of a human labelling a random batch upfront. In domains where labelling is expensive -- radiologists annotating scans, lawyers tagging contract clauses, linguists labelling text for a niche language -- active learning can often reach a target accuracy using 10-30% of the labels a fully-random approach would need.
The active learning loop
The process is iterative: train on whatever labels exist so far, have the model score every remaining unlabelled example by how useful labelling it would be, send the highest-scoring examples to a human for labelling, add the new labels to the training set, and repeat until the model is good enough or the labelling budget runs out.
💡 Key idea: active learning can reach the same accuracy as training on the full dataset while only requiring a fraction of the labels -- often 10-30% in the literature.
Five ways to choose what to label next
1. Uncertainty sampling
Label whatever the model is least confident about. Variants include least-confidence (lowest top-class probability), margin sampling (smallest gap between the top two classes), and entropy sampling (highest predictive entropy across all classes).
2. Query-by-committee
Train several models on the same labelled set and query the examples where they disagree most. Disagreement is often a better uncertainty signal than any single model's confidence score, at the cost of training multiple models.
3. Expected model change
Estimate how much each candidate label would change the model's parameters, and query the examples with the largest expected change. Powerful, but can be computationally expensive to estimate honestly.
4. Expected error reduction
Estimate how much each candidate label would reduce the model's expected future error. In principle optimal; in practice one of the hardest strategies to implement efficiently.
5. Diversity sampling
Cluster the unlabelled pool and query representatives from under-covered clusters, so the labelled set stays spread across the whole feature space rather than clumping around one ambiguous region.
Where it pays off
Active learning earns its keep wherever labelling is the bottleneck: medical imaging, where only specialists can label accurately; NLP for low-resource domains or languages, where annotators are scarce; and computer vision tasks like object detection and segmentation, where a single image can take minutes to annotate correctly.
Practical tips
- Start with a stratified random sample so every class is represented before active selection kicks in.
- Combine uncertainty and diversity rather than relying on either alone -- pure uncertainty sampling can over-focus on one ambiguous region.
- Query in batches, not one example at a time, for efficiency -- but re-score after each batch, since the model changes.
- Monitor a held-out validation curve, not just training accuracy, so you know when to stop labelling.
Tooling
Popular open-source options include modAL (a modular active-learning framework for scikit-learn), ALiPy (a comprehensive active-learning toolbox), and libact (a pool-based active-learning library).
🧪 Try it yourself: the Active Learning Lab simulation lets you experiment with everything described above directly in your browser.