Feature Importance Analysis

Feature Importance Analysis (Verbose)

Core Patterns

The Four Methods Answer Four Questions

"Feature importance" is ambiguous. Name the question before picking the tool.

They routinely disagree, and disagreement is information, not error. A feature high on impurity but flat on permutation was split on often without improving generalization — a classic overfitting signature.

Why Impurity Importance Is Biased

A tree chooses splits by scanning candidate thresholds. A continuous feature with 10,000 distinct values offers 9,999 candidate thresholds; a binary flag offers one. More candidates means more chances that one splits the noise favorably, so the high-cardinality feature accrues impurity reduction it did not earn. The same mechanism promotes free-text hashes, timestamps, and IDs.

import numpy as np
# Control column: pure noise, high cardinality
X_train["_noise"] = np.random.default_rng(0).normal(size=len(X_train))
model.fit(X_train, y_train)

rank = np.argsort(model.feature_importances_)[::-1]
# Anything ranked below "_noise" is not distinguishable from random

This one line reframes every importance table you produce. It is also computed on training data, so a feature that memorizes the training set is rewarded.

Permutation Importance Done Properly

from sklearn.inspection import permutation_importance
import pandas as pd

r = permutation_importance(
    model, X_test, y_test,
    n_repeats=30,
    random_state=0,
    scoring="average_precision",   # match your real evaluation metric
    n_jobs=-1,
)

pd.DataFrame({
    "mean": r.importances_mean,
    "std": r.importances_std,
}, index=X_test.columns).sort_values("mean", ascending=False)

Three rules:

1. Held-out data. Permuting on the training set measures memorization. The sklearn docs are explicit about this; run it on test or in CV.
2. The metric matters. Permutation importance under accuracy and under average precision produce different rankings on imbalanced data — for the same reason those metrics disagree (see model-evaluation).
3. Report the spread. importances_std over n_repeats tells you whether a gap is real. A mean of 0.004 with std 0.006 is zero.

The Correlation Problem

This is the failure senior practitioners watch for. Suppose height_cm and height_in are both present. The model uses whichever it prefers; permuting one leaves the information intact via the other, so both score near zero. A naive reading concludes height does not matter and drops both — destroying a real signal.

There is a second, subtler defect: permutation destroys the joint distribution. Shuffling income independently of loan_amount produces rows with a $20k income and a $2M mortgage. Those rows never existed in training, the model's behavior there is unconstrained extrapolation, and the resulting score drop measures fantasy as much as reliance.

Mitigations, in order of preference:

from scipy.cluster import hierarchy
from scipy.stats import spearmanr
from collections import defaultdict

corr = spearmanr(X_train).correlation
corr = (corr + corr.T) / 2                       # enforce symmetry
np.fill_diagonal(corr, 1)
dist = 1 - np.abs(corr)
links = hierarchy.ward(hierarchy.distance.squareform(dist, checks=False))
cluster_ids = hierarchy.fcluster(links, 0.5, criterion="distance")

groups = defaultdict(list)
for idx, cid in enumerate(cluster_ids):
    groups[cid].append(X_train.columns[idx])
# Permute each group as a unit, or keep one representative per cluster

Alternatives: conditional permutation (permute within strata of the correlated partner), or grouped drop-column, where you refit without the whole cluster.

Drop-Column for Removal Decisions

Permutation answers "does this model use it." If your actual decision is "can we retire this upstream pipeline," only refitting answers that.

from sklearn.model_selection import cross_val_score

base = cross_val_score(pipe, X, y, cv=cv, scoring="average_precision")
for col in candidates:
    without = cross_val_score(pipe, X.drop(columns=[col]), y, cv=cv,
                              scoring="average_precision")
    delta = base.mean() - without.mean()
    se = np.sqrt(base.std()**2 + without.std()**2) / np.sqrt(len(base))
    print(f"{col:<24} delta {delta:+.4f}  se {se:.4f}")

Compare the delta against the interval. Most single-feature removals land inside the noise band, which is itself the answer: the feature is not carrying weight.

Importance Is Not Causation, and Not Fairness Clearance

A high-importance feature may be a proxy. A model that leans on zip_code is often leaning on a correlate of race or income; removing the column does not remove the effect, because the remaining features reconstruct it. Conversely, a protected attribute with low importance does not certify a fair model. Importance ranks model reliance, nothing more. For attribution of individual predictions and its caveats, see the model-interpretability skill.

Common Anti-Patterns

❌ Shipping feature_importances_ as the answer. Biased toward high-cardinality columns and computed on training data. ✅ Permutation importance on held-out data, with a noise-column control.

❌ Permutation importance on the training set. Measures memorization. ✅ X_test / y_test, or importance computed within CV folds.

❌ A ranking with no uncertainty. "0.012 beats 0.009" across 30 shuffles with std 0.008 is not an ordering. ✅ Report importances_mean ± importances_std and treat overlaps as ties.

❌ Pruning correlated features one at a time. Each looks droppable while its twin is present; drop both and lose the signal. ✅ Cluster on rank correlation, evaluate and prune per cluster.

❌ Ignoring an ID-like column at the top of the list. That is a leakage alarm, not a discovery. ✅ Investigate provenance — such a feature almost always encodes the label.

❌ Selecting features on the full dataset, then cross-validating. Selection saw the test folds; scores are optimistic. ✅ Put selection inside the Pipeline so it refits per fold.

❌ Treating low importance for a protected attribute as a fairness result. ✅ Measure outcome disparities directly; proxies survive column removal.

Feature Importance Checklist

• Question named: model reliance vs removal value vs contribution size
• Noise control column added and its rank position noted
• Permutation importance computed on held-out data
• scoring matches the metric the model is actually judged on
• Means reported with standard deviations; overlaps treated as ties
• Correlated features clustered and handled as groups
• Impurity results cross-checked against permutation before use
• Removal decisions confirmed by a refit, not by ranking alone
• Top-ranked features audited for leakage and hindsight
• Rankings verified stable across seeds, folds, and bootstrap resamples
• Feature selection nested inside cross-validation, not applied before it
• No causal or fairness claims drawn from importance alone