φᵢ = xᵢ · ∂p/∂xᵢ, normalized so Σ|φᵢ| = 1
A small credit-approval network (6 input features → 8 → 8 hidden neurons → 2 outputs, Approve/Deny) runs live in your browser. Every glowing sphere is a neuron; every line is a weighted connection. This lab visualizes three real interpretability techniques on top of that network: global feature importance, local gradient-based saliency, and an attention-style flow of "where the explanation travels".
Feature Importance (Global) averages the magnitude of each input's outgoing weights — a common weight-based proxy for how much a feature matters across all possible inputs, independent of any single sample.
Saliency: Gradient × Input (Local) is a genuine backpropagation pass: the network computes ∂p/∂xᵢ, the exact gradient of the Approve probability with respect to each input, then multiplies by the input value itself (the standard "Gradient × Input" attribution method used in real deep-learning interpretability tools):
φᵢ = xᵢ · ∂p/∂xᵢ
The Explanation entropy stat is the Shannon entropy of the normalized |φᵢ| distribution, scaled to 0–100%. Low entropy means one or two features dominate the decision (a concentrated, easy-to-trust explanation); high entropy means the decision is spread thinly across many features (harder to summarize in one sentence).
Explanation method (Feature Importance / Saliency / Attention Flow), feature to perturb, feature value slider, new random sample, play/pause the attention animation
"Gradient × Input" and its more robust cousins (Integrated Gradients, SHAP, LIME) are exactly how tools like Captum and SHAP explain real production models — from credit scoring to medical diagnosis — and are increasingly required by regulations such as the EU's GDPR "right to explanation".
A live 6→8→8→2 neural network explains its own credit-approval decision in real time — switch between global feature importance, local gradient × input saliency, and an animated attention-style flow, then perturb the input to see the explanation change.
Model interpretability is not one technique but a family of them. This simulation runs a real (small) neural network with an actual backpropagation pass, so the "Saliency" mode is a genuine gradient × input attribution — not a canned animation — while "Feature Importance" shows the simpler, global weight-based view that ignores the current input entirely. Watching both side by side makes the local-vs-global distinction concrete.
Pick an explanation method, choose a feature to perturb and drag its value slider, or hit "New sample" for a fresh random input. Switch to "Attention Flow" to watch particles travel from the most-attributed input feature, through the hidden layers, to the predicted class. Drag to orbit the network and scroll to zoom.
"Gradient × Input" and its more robust cousins (Integrated Gradients, SHAP, LIME) are exactly how real interpretability tools explain production models — from credit scoring to medical diagnosis — and are increasingly required by regulations such as the EU's GDPR "right to explanation".