Model Interpretability

Model Interpretability (Verbose)

Core Patterns

Pick the Explanation That Answers the Question

"Make it interpretable" is four different requests. Separate them before choosing a tool.

The last row belongs to the feature-importance-analysis skill; keep the two questions distinct — "what did the model use" is not "what would I lose if I dropped it."

SHAP in Practice

SHAP assigns each feature a credit for moving the prediction away from a base value, using Shapley values from cooperative game theory. The additivity guarantee is what makes it useful: contributions sum exactly to the output.

import shap

explainer = shap.TreeExplainer(model)      # exact for tree ensembles
sv = explainer(X_test)                     # Explanation: .values, .base_values, .data

shap.plots.waterfall(sv[0])                # single prediction, sorted contributions
shap.plots.beeswarm(sv)                    # global: spread and direction per feature
shap.plots.scatter(sv[:, "tenure"])        # learned response shape for one feature

Two details that trip people up:

• Units. For a binary tree classifier, TreeExplainer works in log-odds by default, so contributions do not sum to a probability. Convert deliberately or present the log-odds honestly.
• Background data. Explainers explain relative to a reference distribution. shap.sample(X_train, 100) or shap.kmeans is standard for the model-agnostic explainers; changing the background changes every attribution, so record which one you used.

Pick the right explainer:

Running KernelExplainer on a random forest is a common waste: it approximates by sampling what TreeExplainer computes exactly.

Partial Dependence and ICE

from sklearn.inspection import PartialDependenceDisplay

PartialDependenceDisplay.from_estimator(
    model, X_test,
    features=["tenure", "monthly_charges", ("tenure", "monthly_charges")],
    kind="both",          # ICE curves plus their average (the PDP)
)

A PDP is an average over the marginal distribution, and averages hide heterogeneity. If half the ICE curves slope up and half slope down, the PDP is flat and reports "no effect" for a feature with a strong, interaction-dependent effect. Always look at kind="both" before believing a flat PDP.

PDPs also extrapolate. Computing dependence on age=25 while holding years_employed=40 fixed evaluates the model on a combination that never occurs, and the output there is unconstrained fantasy. Accumulated Local Effects (ALE) avoids this by using conditional rather than marginal distributions and is the safer default under strong feature correlation.

The Central Caveat: Model, Not World

An explanation is a description of a function you fit. It carries no causal claim.

• Proxies get the credit. If number_of_specialist_visits correlates with disease severity, SHAP attributes the prediction to visits. Acting on that — reducing visits — changes the prediction and nothing about the patient.
• Correlated features share credit arbitrarily. With two near-collinear inputs, the fitted model can split the signal between them in many ways that are equally good on training loss. Refit with a new seed and the attribution can swap. The explanation is stable only to the extent the model is.
• A wrong model is confidently explained. SHAP will produce a crisp attribution for a model with 0.55 AUC. Validate accuracy first (see model-evaluation); interpretation of a bad model is elaborate noise.

If the question is genuinely "what happens if we intervene," you need causal inference — an experiment, or an identification strategy with stated assumptions. No post-hoc attribution method substitutes for it.

Validating an Explanation

# 1. Label-shuffle control: nothing should look important
y_shuffled = rng.permutation(y_train)
control = clone(model).fit(X_train, y_shuffled)
# Any feature with strong attribution here indicates leakage or a pipeline bug

# 2. Stability across seeds
rankings = [top_k_shap(clone(model).set_params(random_state=s).fit(X, y))
            for s in range(5)]
# Disagreement in the top 5 means the ranking is not a finding

# 3. Direct perturbation check
x = X_test.iloc[[0]].copy()
x["tenure"] += 12
model.predict_proba(x)[0, 1] - model.predict_proba(X_test.iloc[[0]])[0, 1]
# Should move in the direction and rough magnitude SHAP claimed

Glassbox Models

When the explanation has to survive a regulator or a clinician, consider making the model itself the explanation: penalized logistic regression, a scorecard, a decision list, or an Explainable Boosting Machine (GA2M). These are additive in shape functions, so the "explanation" is exact by construction rather than a post-hoc approximation that may disagree with the model. The accuracy gap versus a tuned gradient-boosting model on tabular data is often 1-2 points of AUC — frequently a good trade when a wrong-and-unjustifiable decision is the real cost.

Common Anti-Patterns

❌ Reading SHAP causally — "reduce support tickets to reduce churn" when tickets merely signal dissatisfaction. ✅ Present attribution as model behavior; escalate to experiment or causal identification before recommending an intervention.

❌ Explaining before validating. A crisp explanation of a model that barely beats the base rate. ✅ Establish predictive performance first, then explain.

❌ KernelExplainer on a tree model. Slow sampling where an exact polynomial-time algorithm exists. ✅ TreeExplainer for tree ensembles, LinearExplainer for linear models.

❌ A flat PDP read as "no effect" on a model with strong interactions. ✅ kind="both" to expose ICE heterogeneity; ALE when features are correlated.

❌ Ignoring the background dataset. Attributions shift with the reference distribution and the choice goes unrecorded. ✅ Fix, document, and re-use the background sample across runs.

❌ Shipping the top-10 feature list without a stability check. ✅ Recompute across seeds and resamples; report only what holds.

❌ Explanations shown to end users in log-odds labeled as probability. ✅ State the units, or transform explicitly and accept the loss of additivity.

Interpretability Checklist

• Explanation question stated: global vs local, descriptive vs actionable
• Model accuracy validated before any explanation is produced
• Explainer matched to model family (TreeExplainer for trees, etc.)
• Background / reference dataset chosen and recorded
• Output units (log-odds vs probability) stated wherever shown
• PDPs accompanied by ICE curves; ALE used under correlated features
• Attribution stability checked across seeds and resamples
• Label-shuffle control run to catch leakage
• SHAP claims spot-checked by perturbing inputs directly
• Causal language avoided unless backed by a causal design
• Glassbox alternative considered where decisions are contestable
• Explanations regenerated after retraining, not carried forward stale