A backdoor (trojan) attack poisons a small fraction of one class's training samples by stamping them with a hidden trigger pattern and relabelling them to the attacker's target class. The network learns two independent decision paths to the same output — but the poisoned samples still light up the last hidden layer differently from the clean ones, because the trigger is an easy, unnatural shortcut the optimizer happily exploits:
clean: x → f(x) → target class (via real features)
poisoned: x+Δ → f(x+Δ) → target class (via trigger Δ, same label)
Activation Clustering (Chen et al., 2018) is a real, practical defense: take the last hidden-layer activation vector for every sample the model assigns to one class, and cluster them with k-means, k=2. This simulator plots two of those activation coordinates directly as (x, y) — the same k-means math a real defender runs on the full high-dimensional vector, just rendered in the two axes you can see:
assign: c(x) = argmin_{j∈{0,1}} ‖a(x) − μ_j‖²
update: μ_j = mean{ a(x) : c(x) = j } (repeat to convergence)
If a class is backdoored, its activations split into a large clean cluster and a small, tight, well-separated cluster of poisoned samples — even though every sample in both clusters gets the identical final label. The smaller cluster is flagged as suspect and its samples are pulled from the training set (or the model is retrained without them). The convergence panel below the scatter plot replays every k-means iteration so you can see the two centroids walk toward the true cluster centers step by step.
- Poison rate — the fraction of this class's samples carrying the trigger.
- Trigger separation — how far the trigger pushes activations from the clean cluster's centroid; a stealthier trigger (low separation) is harder to detect.
- Clean-activation spread — natural variance of legitimate activations; a noisier clean cluster also makes detection harder.
- Centroid separation ratio — inter-cluster centroid distance divided by the average intra-cluster spread, a simplified silhouette-style score; values well above 1 mean the two clusters are cleanly separable.
Real-world relevance: this exact activation-clustering technique (and successors like spectral-signature and fine-pruning defenses) is used to audit third-party and fine-tuned models for supply-chain backdoors before deployment.
Drag the scatter plot to pan, scroll to zoom.