Adversarial machine learning studies how ML models can be deliberately fooled, and how to make them harder to fool. As AI moves into safety- and security-critical systems -- fraud detection, autonomous vehicles, content moderation -- understanding these vulnerabilities stops being academic and starts being a production requirement.
What is an adversarial attack?
An adversarial attack is a deliberately crafted input, usually a small perturbation of a real example, designed to make a model misclassify it. The perturbation is often small enough to be invisible to a human, yet large enough in the model's decision space to flip the prediction -- and these attacks succeed at a surprisingly high rate against undefended models.
💡 Key idea: adversarial attacks demonstrate that high clean-data accuracy does not imply robustness -- the two are genuinely different properties a model can have independently.
Attack families
White-box attacks
The attacker has full access to the model's architecture and weights. Common methods: FGSM (Fast Gradient Sign Method -- one gradient step in the direction that maximises loss), PGD (Projected Gradient Descent -- an iterated, usually stronger version of FGSM), and the Carlini & Wagner attack (an optimisation-based attack that finds minimal perturbations).
Black-box attacks
The attacker has no access to the model internals. Common methods: transfer attacks (craft an adversarial example against a substitute model and hope it transfers), query-based attacks (probe the target model's outputs repeatedly to estimate a useful perturbation direction), and model stealing (reconstruct a copy of the model via its API responses, then attack the copy with white-box methods).
Defences that actually help
Adversarial training
Train on adversarial examples generated on the fly, alongside clean data, so the model is explicitly pushed to keep a decision margin around every training point. Widely regarded as the most effective single defence, at some cost to clean-data accuracy.
Input preprocessing
Denoising, quantisation and feature squeezing all try to strip the perturbation out of the input before it reaches the model. Useful as a layer of defence, but rarely sufficient alone against an attacker who knows the preprocessing step exists.
Certified defences
Methods like interval bound propagation and randomised smoothing provide mathematical guarantees that no perturbation within a given budget can change the prediction -- stronger than empirical robustness, but typically more conservative and computationally heavier.
Where this matters in practice
Security-critical deployments (fraud detection, malware classification, content moderation) need to assume an adaptive attacker, not just a benign distribution shift. Adversarial testing is also increasingly used as a robustness benchmark during development, independent of any real attacker, simply to find where a model's decision boundary sits too close to real data.
🧪 Try it yourself: the Adversarial ML Lab simulation lets you experiment with everything described above directly in your browser.