SHAP — Fair Credit for Every Feature

SHAP is the most important interpretability method in practice today, because it answers "how much did each feature contribute to this prediction?" with a principled, mathematically fair answer borrowed from game theory. Where LIME gives an intuitive but unstable local explanation, SHAP gives a grounded one with guarantees — and it unifies local and global interpretability into one framework. This post explains the idea behind SHAP without the heavy math, and why it became the field's default.

By Pratik Dhanave · July 13, 2026 · 7 min read

#Explainable AI #SHAP #Shapley Value #Interpretability #Machine Learning

Post 4 covered LIME and its lack of guarantees. This post covers the method built to fix that: SHAP (SHapley Additive exPlanations), which brings a rigorous, decades-old idea from cooperative game theory to feature attribution. It’s the technique most interpretability work now reaches for, so understanding its core idea and its trade-offs is essential.

The game-theory idea: fairly dividing credit

SHAP is built on the Shapley value, a concept from cooperative game theory that solves a seemingly unrelated problem: if a team achieves some payout together, how do you fairly attribute credit to each player? The answer Shapley derived is the unique way to split credit that satisfies a set of fairness axioms — essentially, each player’s credit is their average marginal contribution across all the ways the team could have been assembled.

SHAP maps this directly onto prediction: the “game” is making a prediction, the “players” are the features, and the “payout” is the model’s output for a given instance. The SHAP value of a feature is its fair share of credit for pushing the prediction away from a baseline (the average prediction) — computed as its average marginal contribution across all possible subsets of features. In plain terms: how much does adding this feature’s value change the prediction, averaged over all the contexts of other features?

The beauty is that this isn’t an ad-hoc heuristic like LIME’s local fit — it’s the unique attribution satisfying fairness properties: - Efficiency — the feature contributions add up exactly to the difference between this prediction and the baseline, so the explanation fully accounts for the prediction (nothing unexplained). This is the “additive” in SHAP. - Symmetry — features that contribute equally get equal credit. - Dummy — a feature that never changes the prediction gets zero credit. - Consistency — if a model changes so a feature contributes more, its SHAP value won’t decrease (the property that impurity importance violated, post 3).

These guarantees are exactly what LIME lacked (post 4), and they’re why SHAP explanations are stable (not randomly different on reruns) and theoretically trustworthy in a way LIME’s aren’t. SHAP answers “why this prediction?” with a principled, reproducible breakdown of each feature’s contribution.

What SHAP gives you — local and global unified

SHAP’s practical power is that one framework delivers both levels of interpretability (post 1):

Local explanations — for any single prediction, SHAP shows exactly how much each feature pushed the output up or down from the baseline, adding up to the final prediction. “This loan was denied: debt pushed the score down 0.3, income up 0.1, recent missed payment down 0.25…” — a complete, additive, per-feature account of this decision. This is the gold-standard local explanation.
Global explanations — aggregate SHAP values across many instances (e.g. average their magnitudes) and you get a global feature importance that’s grounded in the same principled per-prediction contributions, and that satisfies consistency (unlike impurity importance, post 3). So global importance and local explanation come from one coherent framework, consistent with each other — a major advantage over using separate, possibly contradictory methods.
Rich visualizations — SHAP’s tooling produces widely-used plots: summary plots (global importance + how feature values affect predictions), force plots and waterfall plots (a single prediction’s feature contributions), and dependence plots (how a feature’s effect varies with its value). These make SHAP both rigorous and communicable.

This local-global unification, plus the fairness guarantees, is why SHAP became the de facto standard: it’s the one method that gives trustworthy, consistent explanations at both levels with excellent tooling.

The catch: cost, and what SHAP still can’t do

SHAP isn’t free, and using it well means knowing its limits:

Computational cost. Computing exact Shapley values requires considering all subsets of features, which is exponential — infeasible beyond a handful of features. SHAP is practical only because of approximations: TreeSHAP computes exact values efficiently for tree-based models (a major reason SHAP is so popular on tabular/gradient-boosted models — it’s fast there), while KernelSHAP approximates for any model but is slow and itself approximate. So SHAP is cheap for trees and expensive/approximate for arbitrary black boxes — a key practical consideration.
Correlated features muddy interpretation. Like all attribution methods, SHAP’s handling of correlated features involves assumptions (about how to treat feature subsets when features aren’t independent), and different SHAP variants make different choices, so correlated features can still produce debatable attributions. The fairness guarantees are about the attribution math, not about resolving feature correlation.
It explains the model, not the world — still not causal. This bears repeating (post 3, and the causal inference series): a feature’s SHAP value is its contribution to the model’s prediction, not evidence that the feature causes the outcome or that changing it would help. SHAP’s rigor is about fair attribution within the model, and it’s easy to over-read a clean SHAP plot as a causal story. It isn’t one.
Faithful to the model, including the model’s flaws. SHAP faithfully explains what the model does — so if the model learned a spurious or biased pattern, SHAP will faithfully attribute to it. That’s actually a feature (it reveals the flaw, post 1’s debugging value), but it means a clean-looking SHAP explanation of a bad model is still explaining a bad model.

Used with these in mind, SHAP is the best general-purpose interpretability tool available: reach for it (TreeSHAP especially) when you want stable, principled, local-and-global explanations with good tooling; accept its cost on non-tree black boxes; and never mistake its fair attribution for a causal claim. It fixed LIME’s biggest weaknesses (instability, lack of guarantees) by grounding explanation in game theory, which is why it’s the method most practitioners now default to — while remembering that even the best explanation is still a faithful description of the model, flaws and all, not the truth about the world (post 8).

The throughline: SHAP brings the game-theoretic Shapley value — the unique fair way to divide credit among cooperating players — to feature attribution: the “players” are features, the “payout” is the prediction, and a feature’s SHAP value is its average marginal contribution across all feature subsets. Unlike LIME’s ad-hoc local fit, this satisfies fairness axioms — efficiency (contributions sum exactly to prediction minus baseline — the “additive”), symmetry, dummy, consistency — giving stable, principled, reproducible explanations and fixing LIME’s biggest flaws. One framework unifies local (per-prediction additive breakdowns — the gold standard) and global (consistent aggregated importance) interpretability with rich visualizations, which is why it’s the de facto standard. The catch: exact Shapley values are exponential, so it relies on approximations (TreeSHAP fast/exact for trees — why it’s popular on tabular data; KernelSHAP slow/approximate for any model); correlated features still muddy attribution; and critically it explains the model, not the world — a SHAP value is fair attribution within the model, not a causal claim, and it faithfully explains flawed models too.

Key takeaways

SHAP grounds feature attribution in the Shapley value from cooperative game theory — the unique fair way to split credit — treating features as “players”, the prediction as the “payout”, and a feature’s SHAP value as its average marginal contribution across all feature subsets.
It satisfies fairness axioms LIME lacked: efficiency (contributions sum exactly to prediction − baseline — the “additive”/complete account), symmetry, dummy (no-effect feature gets zero), and consistency (more contribution never lowers the value — which impurity importance violated) — making explanations stable and principled, not randomly different on reruns.
One framework unifies local (a per-prediction additive breakdown of each feature’s push up/down — the gold-standard local explanation) and global (consistent aggregated importance) interpretability, with widely-used visualizations (summary, force/waterfall, dependence plots) — why it’s the de facto standard.
The catch is cost: exact Shapley values are exponential, so SHAP relies on approximations — TreeSHAP (fast & exact for tree models — a big reason SHAP dominates on tabular/boosted models) and KernelSHAP (any model, but slow and approximate); correlated features still complicate attribution.
It explains the model, not the world: a SHAP value is fair attribution within the model, not causal (don’t read a clean SHAP plot as “changing X helps” — causal inference series), and it faithfully explains flawed models too (revealing their biases — useful for debugging, but a tidy SHAP plot of a bad model still explains a bad model). Best general-purpose tool, used with these limits.