This scene shows a small feed-forward neural network — like the ones used for loan approval, fraud flags or medical triage — deciding whether to Approve or Deny an applicant. Six input features flow through two hidden layers to an output layer. The network itself gives no explanation for its answer; it is a black box unless we probe it.
Real SHAP (SHapley Additive exPlanations) values come from cooperative game theory — they average a feature's marginal contribution over every possible ordering of features, which is expensive. This simulation uses a single-pass occlusion approximation so the idea stays visible in real time, but the same "remove it and see what changes" logic is at the heart of both.
A small feed-forward network decides whether to approve or deny a loan applicant. Feeding in different sample inputs recomputes real occlusion-based feature attributions — a SHAP-adjacent technique — and lights up the exact nodes and edges that pushed the decision toward Approve or Deny.
Each node's glow is computed by zeroing that node and re-running the network downstream, then measuring how much the output probability changed — the same "remove it and see what happens" logic behind SHAP and occlusion sensitivity.
Pick an applicant sample or drag the Income / Debt ratio sliders to build a custom profile. Raise the attribution threshold to strip away weak connections and isolate only the strongest reasoning paths, then replay the forward pass to watch the signal travel layer by layer.
True SHAP values average a feature's marginal contribution across every possible ordering of features — computationally expensive for large models, which is why fast approximations like this one are common in practice.