Explainable AI: Opening the Black Box of Model Decisions
How feature attribution methods like SHAP and LIME work, why inherently interpretable models sometimes beat post-hoc explanations, and where explainability and accuracy genuinely trade off.
Why a correct prediction is not the same as a trusted one
A model that denies someone a loan, flags a tax return for audit, or recommends against parole can be statistically excellent and still be operationally unusable if nobody — not the applicant, not the compliance officer, not the engineer debugging a regression — can say why it produced that particular output for that particular case. This is the gap explainable AI (XAI) exists to close. It is worth being precise about what "explanation" means here, because the term covers at least two distinct goals that often get conflated: global interpretability (understanding what the model has learned in general, across its whole input space) and local explainability (understanding why the model made this specific decision for this specific input). A credit model might be globally dominated by income and debt-to-income ratio, yet for one rejected applicant the deciding factor was a recent missed payment that barely moves the global feature-importance ranking. Debugging, auditing and individual-rights compliance (the EU GDPR's "right to explanation" being the most cited legal driver) almost always need the local version.
The practical stakes of getting this right showed up starkly in Chicago's COMPAS recidivism tool and Amazon's abandoned resume-screening model, both cases where a model's aggregate accuracy looked acceptable but post-hoc analysis revealed systematic, protected-characteristic-correlated errors that nobody had surfaced during development because nobody had a working method for asking the model "why did you decide this?" case by case. Explainability, in other words, is not merely a nice-to-have transparency feature; it is often the only practical mechanism for catching a certain class of bug — the model that is right on average and wrong in a systematically harmful pattern.
Feature attribution: SHAP and LIME
The two dominant post-hoc explanation techniques, LIME and SHAP, solve the local explanation problem by different routes, and understanding the difference matters for choosing between them. LIME (Local Interpretable Model-agnostic Explanations) works by perturbation: around the specific input you want explained, it generates a cloud of nearby synthetic samples (for tabular data, jittering feature values; for text, dropping words; for images, occluding superpixels), gets the black-box model's predictions on each, and then fits a simple, interpretable model — typically a sparse linear regression — to that local neighbourhood of predictions. The coefficients of that local linear model become the explanation: "in the vicinity of this particular applicant, income contributed +0.3 to the approval score and recent missed payment contributed -0.5." LIME is fast and model-agnostic but its explanations are only as stable as the neighbourhood sampling; run it twice with different random perturbations and you can get noticeably different attributions for the same prediction, which is a real weakness in high-stakes settings.
SHAP (SHapley Additive exPlanations) takes a more principled route, borrowing from cooperative game theory. It treats each feature as a "player" contributing to the "payout" (the prediction), and computes each feature's Shapley value — its average marginal contribution across every possible ordering in which features could be "added" to a baseline prediction. This guarantees a set of desirable mathematical properties that LIME does not: the attributions for all features sum exactly to the difference between the model's prediction and its average baseline prediction (local accuracy), and a feature that never affects the output gets exactly zero credit (missingness). The cost is computational: exact Shapley values require evaluating the model on an exponential number of feature subsets, which is why practical implementations use approximations — TreeSHAP exploits the internal structure of gradient-boosted trees for a fast exact-ish computation, while KernelSHAP falls back to a LIME-like sampling approach for arbitrary models, trading some of the theoretical guarantee for tractability on deep networks.
Inherently interpretable models: the other path
Post-hoc explanation is not the only strategy, and for some problems it is the wrong one. The alternative is to constrain the model class itself to something a person can read directly: a shallow decision tree, a linear or logistic regression with a modest number of features, a generalized additive model (GAM) that sums independent per-feature curves, or a rule list. Cynthia Rudin's influential argument in this space is that for high-stakes decisions, reaching for a black box and then explaining it after the fact is frequently unnecessary and sometimes actively dangerous, because a post-hoc explanation is a model of the model — an approximation that can be unfaithful in exactly the cases that matter most, while an inherently interpretable model's "explanation" is simply what it computed, with no approximation gap at all. Her team's work replicating COMPAS-level recidivism prediction accuracy with a short, human-readable rule list is the canonical demonstration that the black box was frequently not buying any accuracy in the first place.
That said, inherently interpretable models are a real constraint, not a free lunch: a GAM cannot capture complex feature interactions the way a gradient-boosted tree or neural network can, and for problems where the true underlying relationship is genuinely high-dimensional and non-additive — image recognition, speech, many natural-language tasks — forcing the model into an interpretable shape can cost real accuracy. The pragmatic answer most practitioners land on is: try the interpretable model first, measure the accuracy gap against the best black box on held-out data, and only accept the complexity (and the post-hoc explanation burden that comes with it) if that gap is large enough to justify it for the specific decision at stake.
The trade-off is real, but it is not universal
It has become common to assert that explainability and accuracy trade off, and the honest answer is: sometimes, substantially, and sometimes barely at all, depending on the structure of the underlying problem. On many well-studied tabular datasets — credit scoring, medical risk scores, recidivism — the accuracy gap between a carefully tuned interpretable model and the best black box turns out to be small, often within a percentage point or two of AUC, because the true signal in the data is close to additive and low-dimensional even though it looks complicated at first glance. On problems with genuine high-order feature interactions, dense unstructured inputs, or very large feature spaces — fraud detection with thousands of engineered features, image classification, language modelling — the gap can be large and interpretable models can badly underperform, which is exactly why almost no serious computer vision or NLP system in production today is a decision tree.
The practical implication for a team choosing between the two approaches is to treat the trade-off as an empirical question to measure on their own data and decision, not a rule to assume. A second, often underweighted factor is what the explanation is actually for: if the goal is regulatory disclosure to an individual applicant, a locally faithful post-hoc explanation from SHAP may be entirely sufficient even on top of a black-box model. If the goal is for a domain expert to audit and trust the model's overall reasoning before deployment, an inherently interpretable model, whose "explanation" carries no approximation risk, is usually the safer choice, precisely because in a black box even a highly faithful post-hoc explanation is still an approximation and can mislead in edge cases the perturbation sampling did not cover well.
Frequently Asked Questions
Is SHAP always better than LIME?
SHAP gives mathematically consistent attributions that sum to the model's actual output difference from baseline, which LIME does not guarantee. LIME is faster and simpler to compute, which matters for very large models or real-time explanation needs, but its explanations can vary between runs due to random sampling.
Can a post-hoc explanation ever be wrong about what the model actually did?
Yes. Both LIME and SHAP approximate the black box's local behaviour rather than reading its internal computation directly, so the explanation can diverge from the model's true reasoning, particularly in regions of the input space with sparse or unusual data that the perturbation or sampling process under-represents.
Why not just always use an interpretable model to avoid the problem entirely?
For problems with strong non-additive feature interactions or high-dimensional unstructured inputs like images and text, constraining the model to an interpretable form can cost meaningful accuracy, which may not be an acceptable trade-off depending on the application.
What is the difference between global and local interpretability?
Global interpretability describes what the model has learned overall, across the entire dataset, such as which features matter most on average. Local interpretability explains a single prediction for a single input, which is usually what regulatory rights to explanation and case-by-case debugging actually require.