A backdoor (trojan) attack poisons a small fraction of one class's training samples by stamping them with a hidden trigger pattern and relabelling them to the attacker's target class. The network learns two independent decision paths to the same output — but the poisoned samples still light up the last hidden layer differently from the clean ones, because the trigger is an easy, unnatural shortcut the optimizer happily exploits:
clean: x → f(x) → target class (via real features)
poisoned: x+Δ → f(x+Δ) → target class (via trigger Δ, same label)
Activation Clustering (Chen et al., 2018) is a real, practical defense: take the last hidden-layer activation vector for every sample the model assigns to one class, and cluster them with k-means, k=2:
assign: c(x) = argmin_{j∈{0,1}} ‖a(x) − μⱼ‖²
update: μⱼ = mean{ a(x) : c(x) = j } (repeat to convergence)
If a class is backdoored, its activations split into a large clean cluster and a small, tight, well-separated cluster of poisoned samples — even though every sample in both clusters gets the identical final label. The smaller cluster is flagged as suspect and its samples are pulled from the training set (or the model is retrained without them).
- Poison rate — the fraction of this class's samples carrying the trigger.
- Trigger separation — how far the trigger pushes activations from the clean cluster's centroid; a stealthier trigger (low separation) is harder to detect.
- Clean-activation spread — natural variance of legitimate activations; a noisier clean cluster also makes detection harder.
- Centroid separation ratio — inter-cluster centroid distance divided by the average intra-cluster spread, a simplified silhouette-style score; values well above 1 mean the two clusters are cleanly separable.
Real-world relevance: this exact activation-clustering technique (and successors like spectral-signature and fine-pruning defenses) is used to audit third-party and fine-tuned models for supply-chain backdoors before deployment.