What does the 'coverage' stat in this simulation mean?

It compares the sum of the four single-feature attributions against the true total change in output versus the neutral baseline. Anything below 100% is the network's non-linear interaction effects — the part no single-feature attribution method can fully capture.

Why does the black box container matter?

It visualises the core problem this simulation is about: a deep network's hidden layers are, by default, opaque to a human observer. Attribution methods like the one shown here don't open the box physically — they infer influence from input/output behaviour alone.

Home▸Machine Learning & Neural Networks▸AI Explainability: Understanding the Black Box

🧠 AI Explainability: Understanding the Black Box

A live feed-forward neural network you can interrogate: drag input-feature sliders and watch a simplified Shapley-value attribution engine highlight exactly which feature pushed the black-box decision, and by how much.

Machine Learning & Neural Networks3DModerate60 FPS

How it Works

The network is a real 4→6→4→1 feed-forward network with fixed, hand-seeded weights — it takes four normalised loan-application features and outputs an approval probability through two tanh hidden layers and a sigmoid output. In "black box" mode you only see the input sliders and the final probability; a solid dark shell hides everything in between, exactly like a production model behind an API.

Pressing "Reveal explanation" doesn't just make the shell transparent — it runs a genuine (if simplified) attribution pass: for every feature, the engine reruns the same network with only that feature swapped back to a neutral baseline of 0.5, and measures how much the output moves. That movement is the feature's attribution, colour-coded green (pushed toward approval) or red (pushed toward denial) on the edges leaving that input node.

Single-feature attribution (first-order Shapley approximation): φᵢ ≈ f(x) − f(x with feature i → baseline) Full Shapley value averages that swap over every coalition S of the other features: φᵢ = Σ_S⊆F∖{i} [|S|!·(n−|S|−1)!]/n! · [f(S∪{i}) − f(S)] This demo computes only the single term above (n = 4, one baseline swap per feature) — the same first-order idea LIME and SHAP build on, without the 2ⁿ-subset averaging a full Shapley computation requires.

Frequently Asked Questions

What's the difference between AI Explainability and Interpretability?

Interpretability focuses on understanding a model's internal workings, while explainability is about providing justifications for specific predictions.

Can I fully trust an XAI system?

XAI systems provide insights, but they are not infallible. It's crucial to critically evaluate the explanations and consider potential biases.

How does XAI relate to fairness in AI?

By identifying biased features or decision-making patterns, XAI can help ensure that AI systems treat different groups fairly.

What does the "coverage" stat mean?

It compares the sum of the four single-feature attributions against the network's actual total output change versus the neutral baseline. Anything below 100% is non-linear interaction between features — the part no single-feature attribution method fully captures.

Why does the black box shell matter here?

It visualises the core problem this simulation is about: a deep network's hidden layers are, by default, opaque. Attribution methods like the one shown don't physically open the box — they infer influence purely from input/output behaviour.

About this simulation

This simulator runs a real 4→6→4→1 feed-forward network behind a loan-approval decision. In "black box" mode the hidden layers sit inside an opaque shell — you only ever see the four inputs and the final probability, just like calling a production model through an API. Press "Reveal explanation" and the shell turns near-transparent while a simplified Shapley-style attribution pass runs: each input feature is swapped, one at a time, back to a neutral baseline, and the resulting shift in output becomes that feature's colour-coded contribution.

🔬 What it shows

A live forward pass through tanh/sigmoid layers, a black-box shell that can be toggled open, and a genuine single-feature attribution computation — not a canned animation — recomputed from the actual network weights every time a slider moves.

🎮 How to use

Drag Income, Credit score, Debt ratio and Years employed. Click "Reveal explanation" to open the box and colour the input edges by attribution (green = pushes toward approval, red = pushes toward denial). Reset inputs returns every slider to the neutral 0.50 baseline.

💡Did you know?

The "attribution coverage" stat rarely reaches 100% — the gap is the network's non-linear feature interactions, the exact reason real explainability tools like SHAP have to average over every possible feature coalition instead of testing features one at a time.

Frequently asked questions

Why does moving one slider sometimes change another feature's attribution colour?

Because the network is non-linear, a feature's marginal contribution f(x) − f(x with i→baseline) depends on the values of every other feature at that moment — attribution isn't a fixed weight per feature, it's computed fresh for the current input vector every time.

Why is "attribution coverage" almost never exactly 100%?

Summing the four single-feature swaps only captures first-order effects. The network's hidden layers also encode interaction effects — e.g. income mattering more when debt ratio is low — that a one-feature-at-a-time test structurally cannot see, which is exactly why full Shapley values average over 2⁴ feature coalitions instead of 4.

What decides whether a feature's edges turn green or red?

The sign of φᵢ = f(x) − f(x with i→baseline): a positive value means the current slider value pushed approval probability up relative to the neutral 0.5 baseline, a negative value means it pushed it down.

Why does the black box shell stay solid until I click Reveal?

It's a deliberate stand-in for a real deployed model: from outside an API you only ever see inputs and outputs. The attribution pass that runs on Reveal doesn't peek at the hidden-layer activations directly either — it infers influence purely by testing input variations, same as real SHAP/LIME tooling.

Is this simulation using real SHAP values?

No — it computes the single-swap term φᵢ ≈ f(x) − f(x_{i→baseline}), a fast first-order approximation. True Shapley values average that swap over every possible subset of the other features (2ⁿ subsets), which is exponential and impractical to animate live in a browser for a teaching demo.

⚙ Under the hood

A real 4-6-4-1 feed-forward network scores a loan application behind an opaque black-box shell. Reveal it to run a simplified Shapley-value attribution pass — phi_i = f(x) - f(x with feature i swapped to baseline) — that colours each input feature's contribution to the decision.

Three.jsAI explainabilityXAISHAPLIMEneural networkblack boxmachine learning

3D · Three.js / WebGL renderer · 60 FPS target · runs fully client-side, no install