Interpretability is the ability to understand and explain how a model arrives at its decisions. As AI is used for more consequential decisions -- credit, hiring, medical triage -- interpretability stops being a nice-to-have and becomes a genuine requirement: for trust, for regulatory compliance (GDPR, the EU AI Act), for debugging, for improving the model, and for catching bias before it causes harm.
Two levels of interpretation
Global interpretability
Understanding the model's overall behaviour: which features matter most on average, how the model behaves across the whole population, general patterns and rules it has learned.
Local interpretability
Understanding one specific prediction: why this particular output, which features drove it, what would change if an input were different.
Two families of methods
Intrinsic methods
Models that are naturally interpretable by construction: linear models (coefficients directly show feature importance), decision trees (rules are human-readable), rule-based systems (explicit if-then logic).
Post-hoc methods
Methods applied after training to explain an otherwise opaque model: SHAP (Shapley Additive Explanations), LIME (Local Interpretable Model-agnostic Explanations), partial dependence plots, and feature-importance measures.
💡 Key idea: interpretability is not only a technical nicety -- it is increasingly a precondition for responsible, accountable AI deployment.
SHAP values
SHAP is grounded in cooperative game theory: it treats each feature as a "player" contributing to the prediction and computes a fair distribution of credit (a Shapley value) for each one, averaged across every possible order features could be considered in. It works for any model architecture and supports both local (single prediction) and global (aggregated) interpretation -- at real computational cost, since the exact calculation is expensive and practical implementations approximate it.
LIME
LIME explains individual predictions by fitting a simple, interpretable model (like a linear model) locally around the specific point being explained -- fast and effective for a single prediction, though the local approximation may not generalise to how the model behaves elsewhere.
Reading the visualisations
Feature-importance plots show global behaviour; waterfall plots show a single prediction's contribution breakdown, feature by feature, starting from a base value; partial dependence plots show how the prediction changes as one feature varies with others held fixed; heatmaps visualise more complex feature interactions.
Tooling and best practice
Common libraries: SHAP (the most widely used), LIME, ELI5 (simple interface), InterpretML (Microsoft), Captum (PyTorch-native). Choose a method that matches your model type and audience, and validate that explanations are actually faithful to the model's real behaviour rather than just plausible-looking.
🧪 Try it yourself: the Model Interpretability Lab simulation lets you experiment with everything described above directly in your browser.