Explainable AI and interpretability from first principles — opening the black box, complementing the AI governance and ML series. Why interpretability matters (the accuracy-vs-interpretability trade-off and the black-box problem; why 'accurate but unexplainable' fails — trust, debugging models right for the wrong reasons, fairness, accountability, regulation; global vs local, model-specific vs agnostic), intrinsic vs post-hoc (two paths — transparent-by-design glass-box models giving faithful explanations vs explaining a black box afterward with possibly-unfaithful methods; the argument to prefer intrinsic for high-stakes problems), feature importance (the workhorse global method; permutation importance over biased built-in importances; the traps — correlated features and importance-is-not-causation), LIME (local model-agnostic explanation by locally approximating the model with a simple one; intuitive but unstable and gameable), SHAP (the field's default — game-theoretic Shapley values giving fair, stable, principled attribution that unifies local and global; TreeSHAP's efficiency; still attribution not causation), counterfactual and example-based explanations (what would change the outcome — actionable recourse; and similar-case reasoning — matching how humans explain), interpreting deep networks and LLMs (saliency maps, attention-is-not-explanation, probing, and mechanistic interpretability's partial reverse-engineering — with the honesty that we largely can't yet explain frontier LLMs), and interpretability in practice (matching method to need, the traps causing false confidence, and the skepticism that separates understanding from a rubber stamp). Grounded in Wikipedia (explainable artificial intelligence, black box, decision tree learning, random forest, surrogate model, Shapley value, case-based reasoning, large language model, right to explanation).
Machine learning models increasingly make decisions that affect people's lives — who gets a loan, a diagnosis, a job interview — and many of them are black boxes that can't explain why. Interpretability is the field dedicated to opening those boxes: understanding why a model made a prediction, in terms a human can grasp. As AI gets more powerful and more deployed, being able to explain it stops being a nicety and becomes a requirement. This series builds interpretability from first principles.
The most accurate models are often the least interpretable — black boxes that can't explain why. Why 'accurate but unexplainable' isn't enough (trust, debugging models that are right for the wrong reasons, fairness, accountability, regulation), and the questions interpretability actually asks (global vs local, model-specific vs agnostic).
There are exactly two ways to get an explainable model: build one that's transparent from the start, or build a black box and explain it afterward. These two paths — intrinsic interpretability and post-hoc explanation — define the entire field, and choosing between them is the first real decision in any project where understanding matters. This post lays out both, their trade-offs, and why the choice is less obvious than "just use the accurate model and explain it later."
Two paths to an explainable model: build one transparent by design (intrinsic — faithful, free explanations) or explain a black box afterward (post-hoc — keeps accuracy but explanations can be unfaithful). The real debate: for high-stakes problems where an interpretable model is competitive, prefer intrinsic over explaining a black box.
The most basic and most used interpretability question is also the most useful: which features actually matter to this model? Feature importance answers it, giving a ranked list of the inputs that drive a model's predictions — the first thing to compute when you want to understand, debug, or trust any model. This post covers the main ways to measure it, the traps that make naive importance misleading, and why permutation importance is the reliable default.
Which features actually matter? Feature importance is the workhorse global method — great for sanity-checking and leakage detection — but built-in importances mislead (impurity bias), so prefer permutation importance. And the traps: correlated features hide importance, and importance is not causation.
Global feature importance tells you what a model relies on overall, but the question people actually ask is local: why did the model deny this particular loan? LIME answers it with a clever trick — approximate the complicated model with a simple, interpretable one in the small neighborhood around a single prediction. It was the method that popularized model-agnostic local explanations, and understanding how it works (and where it's fragile) is essential interpretability literacy.
Why did the model make this one prediction? LIME approximates the complex model with a simple one in the small neighborhood around a single instance — intuitive, model-agnostic, launched the modern wave of local explanation. But it's fragile: unstable across reruns, sensitive to choices, and gameable — which is why SHAP followed.
SHAP is the most important interpretability method in practice today, because it answers "how much did each feature contribute to this prediction?" with a principled, mathematically fair answer borrowed from game theory. Where LIME gives an intuitive but unstable local explanation, SHAP gives a grounded one with guarantees — and it unifies local and global interpretability into one framework. This post explains the idea behind SHAP without the heavy math, and why it became the field's default.
SHAP is the field's default: it answers 'how much did each feature contribute?' with a fair, game-theoretic answer (Shapley values), fixing LIME's instability with real guarantees and unifying local and global interpretability. The catch: exponential cost (TreeSHAP fixes it for trees), and it's attribution, not causation.
Feature attributions tell you which inputs mattered, but humans often understand decisions better through a different lens: "what would have to change for a different outcome?" and "what similar cases did the model see?" Counterfactual and example-based explanations answer exactly those questions — and they're often more actionable and more intuitive than attribution scores. This post covers the explanation styles that speak the way people actually reason.
Humans understand decisions by contrast ('what would change the outcome?') and example ('what similar cases?'). Counterfactual explanations give actionable recourse ('approved if income were $5k higher'); example-based explanations build trust by analogy. No single explanation is right — match the style to the question and audience.
The hardest interpretability frontier is the models that matter most right now: deep neural networks and large language models, with billions of parameters and no obvious place to look for "the reason." The attribution methods that work on tabular models strain here, and a new field — mechanistic interpretability — is trying to reverse-engineer what these networks actually compute. This post covers how we peer inside deep models and LLMs, and why it's both crucial and genuinely unsolved.
The hardest frontier: deep nets and LLMs with billions of parameters and no obvious place to find 'the reason'. Saliency maps (fragile), attention (not explanation), probing, and mechanistic interpretability's attempt to reverse-engineer circuits — plus the honest truth that we largely cannot yet explain why frontier LLMs behave as they do.
Interpretability has a toolbox — intrinsic models, feature importance, LIME, SHAP, counterfactuals, saliency maps — but knowing the tools isn't the same as using them wisely. The real skill is matching the method to the need, reading explanations with appropriate skepticism, and avoiding the traps that turn interpretability into false confidence. This closing post is the practical judgment the series was building toward: how to actually do interpretability without fooling yourself.
Knowing the tools isn't using them wisely. Matching the method to the need (model, level, audience), the traps that cause false confidence (explanations aren't truth, explanation isn't causation, plausible isn't correct, interpretability-washing), and the skepticism that separates genuine understanding from a rubber stamp.
This series is part of a larger body of work by Pratik Dhanave, an Agentic AI Architect writing about production AI systems, distributed systems, and cloud-native engineering. Explore all course series, browse every post, or find topics via the tag index.