Explainable AI: Explaining One Prediction Without Explaining the Whole Model

Explaining one decision, not the whole model

Many high-performing models — deep networks, gradient-boosted trees, kernel methods — are globally too complex to summarize in a sentence. Local surrogate methods, of which LIME (Local Interpretable Model-agnostic Explanations) is the best-known, sidestep that entirely by not trying to explain the whole model at once. Instead, they explain one specific prediction by fitting a simple, interpretable model — usually linear — that mimics the complex model's behaviour only in a small neighbourhood around that one point.

How the local fit actually works

To explain a prediction, LIME-style methods sample points near it, weight each sample by proximity (typically a Gaussian kernel on distance), and fit a weighted linear regression to the true model's outputs over that neighbourhood. The resulting coefficients become the explanation: how much the prediction would locally change per unit increase in each feature, right around the point being explained.

Why the same feature can matter differently elsewhere

A nonlinear model can behave completely differently in different regions of its input space, so a single global linear summary would be actively misleading almost everywhere it was applied. Explaining locally is the trade-off local surrogates make explicit: an honest, human-readable explanation of one decision, at the cost of that explanation not necessarily transferring to any other decision the same model makes elsewhere.

LIME vs. SHAP

SHAP (SHapley Additive exPlanations) is a related but mathematically distinct approach: instead of fitting an approximate local model, it computes each feature's exact contribution to a specific prediction using game-theoretic Shapley values. Both target the same goal — local, per-prediction interpretability — from different mathematical foundations, and both are standard tools in a production explainability toolkit.