As AI moves into safety- and business-critical systems, security stops being a bolt-on concern. This guide covers the attack surface beyond a single misclassified input: how attackers with different levels of access operate, and which defenses actually hold up.
Attack families by attacker access
White-box attacks (full model access)
FGSM (a single gradient step toward higher loss), PGD (an iterated, usually stronger version of FGSM), and the Carlini & Wagner attack (an optimisation-based search for a minimal perturbation).
Black-box attacks (no internal access)
Transfer attacks (craft against a substitute model, hope it transfers), query-based attacks (probe the target repeatedly to estimate a useful direction), and model stealing / extraction (reconstruct a working copy of the model from its API responses alone).
💡 Key idea: a model doesn't need to be attacked directly to be compromised -- a well-resourced attacker can build a working copy of it just by querying its public API enough times.
Model extraction: stealing without touching the weights
An attacker with only API access can send many queries, observe the outputs, and train a substitute model on those input-output pairs -- no access to architecture or parameters required. The stolen model won't be a perfect copy, but it can be good enough to run further white-box attacks against, or to sidestep paying for the original service entirely.
Defenses that hold up
Adversarial training
Train on adversarial examples alongside clean data so the decision boundary keeps a margin from every training point. Widely considered the strongest single defense against evasion attacks, at some cost to clean-data accuracy.
Input preprocessing
Denoising, quantisation and feature squeezing strip perturbations before they reach the model -- useful as a layer, rarely sufficient alone against an attacker who knows the preprocessing exists.
Certified defenses
Interval bound propagation and randomised smoothing provide mathematical guarantees against perturbations within a bounded budget -- stronger than empirical robustness, typically more conservative and computationally heavier.
Defending against extraction specifically
Rate-limiting reduces the volume of usable query-response pairs an attacker can gather; returning rounded confidence scores or label-only output reduces the information content of each individual response. None of these make extraction impossible -- they raise its cost and reduce its fidelity.
Where this matters most
Fraud detection, malware classification and content moderation all need to assume an adaptive attacker, not a benign distribution shift. Model extraction concerns apply to any API-exposed model, public or internal, that a motivated party can query repeatedly.
🧪 Try it yourself: the AI Security Lab simulation lets you experiment with everything described above directly in your browser.