LoopyLuci Interpretable ML
Explaining and interpreting machine learning model predictions.
When to Use Each Method
| Method | Scope | Speed | Model-Agnostic | Best For |
|---|---|---|---|---|
| SHAP | Local + Global | Slow | Yes | Any model, feature importance |
| LIME | Local | Medium | Yes | Single prediction explanation |
| Integrated Gradients | Local | Fast | No | Deep learning |
| Attention Weights | Local | Fast | No | Transformers |
| Partial Dependence | Global | Medium | Yes | Feature effect |
| Permutation Importance | Global | Fast | Yes | Feature ranking |
SHAP
import shap
# Tree explainer (fast for XGBoost, LightGBM, RF)
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X)
# Summary plot
shap.summary_plot(shap_values, X)
# Force plot for single prediction
shap.force_plot(explainer.expected_value, shap_values[0], X.iloc[0])
# Deep explainer (for neural networks)
explainer = shap.DeepExplainer(model, background_data)
shap_values = explainer.shap_values(X_test[:100])
LIME
from lime import lime_tabular
import numpy as np
explainer = lime_tabular.LimeTabularExplainer(
X_train.values,
feature_names=feature_names,
class_names=['negative', 'positive'],
mode='classification',
)
exp = explainer.explain_instance(
X_test.iloc[0].values,
model.predict_proba,
num_features=10,
)
exp.show_in_notebook()
exp.as_list() # [(feature, weight), ...]
Integrated Gradients
def integrated_gradients(model, input_tensor, baseline=None, steps=50):
if baseline is None:
baseline = torch.zeros_like(input_tensor)
scaled_inputs = [baseline + (i/steps) * (input_tensor - baseline)
for i in range(steps + 1)]
scaled_inputs = torch.stack(scaled_inputs)
scaled_inputs.requires_grad_(True)
outputs = model(scaled_inputs)
pred = outputs[:, outputs.argmax(dim=-1)]
grad = torch.autograd.grad(pred.sum(), scaled_inputs)[0]
avg_grad = (grad[:-1] + grad[1:]).mean(dim=0) / 2
ig = (input_tensor - baseline) * avg_grad
return ig.sum(dim=-1) # feature attributions per token
Attention Visualization
# For transformer models
def visualize_attention(tokens, attention_weights, layer=-1, head=0):
# attention_weights: (layers, heads, seq_len, seq_len)
attn = attention_weights[layer, head].detach().numpy()
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(8, 8))
ax.imshow(attn, cmap='Blues')
ax.set_xticks(range(len(tokens)))
ax.set_yticks(range(len(tokens)))
ax.set_xticklabels(tokens, rotation=90)
ax.set_yticklabels(tokens)
Pitfalls
- SHAP is computationally expensive for large models and datasets
- LIME results vary between runs due to sampling — set seed for reproducibility
- Attention ≠ explanation (different attention distributions can give same output)
- Feature importance measures correlation, not causation
- Local explanations don't generalize to global behavior