Ml4t Feature Validation

Feature Validation

A feature with IC of 0.05 on the full sample may have IC of 0.12 in one year and -0.03 in every other year. Without validation, the model trains on noise disguised as signal.

The Problem

Skipping feature validation leads to three failures: lookahead contamination, regime-specific features that fail live, and redundant features that waste model capacity.

The Pattern

WRONG

from sklearn.ensemble import GradientBoostingRegressor

# Train on all features without any validation - overfitting guaranteed
model = GradientBoostingRegressor(n_estimators=200)
model.fit(X_train, y_train)  # 50 features, no idea which are noise

CORRECT

from scipy.stats import spearmanr
import numpy as np

def validate_feature(feature: np.ndarray, target: np.ndarray, dates: np.ndarray) -> dict:
    """Screen a single feature for predictive quality."""
    ok = ~np.isnan(feature) & ~np.isnan(target)  # both: one nan makes ic nan
    ic, p_value = spearmanr(feature[ok], target[ok])
    quarters = dates.astype("datetime64[M]").astype(int) // 3  # numpy has no [Q]
    quarterly_ics = []
    for q in np.unique(quarters):
        mask = (quarters == q) & ok  # ok too: the floor counts VALID pairs
        if mask.sum() > 30:
            qic, _ = spearmanr(feature[mask], target[mask])
            quarterly_ics.append(qic)

    quarterly_ics = np.array(quarterly_ics)
    quarterly_ics = quarterly_ics[~np.isnan(quarterly_ics)]  # nan != a bad quarter
    ic_mean, ic_std = quarterly_ics.mean(), quarterly_ics.std()
    ic_ir = ic_mean / ic_std if ic_std > 0 else 0  # IC information ratio
    leakage_flag = abs(ic) > 0.10

    return {"ic": ic, "p_value": p_value, "ic_ir": ic_ir, "leakage_flag": leakage_flag,
            "pct_positive_quarters": np.mean(quarterly_ics > 0)}

Validation Checklist Sequence

IC Decay Analysis

def ic_decay(feature: np.ndarray, returns: np.ndarray, horizons: list[int]) -> dict:
    """IC should decay with horizon - if it doesn't, suspect leakage."""
    n, decay = len(returns), {}
    gaps = np.r_[0, np.cumsum(np.isnan(returns))]  # missing returns so far
    cum = np.r_[1.0, np.cumprod(1.0 + np.nan_to_num(returns))]
    for h in horizons:
        if not 0 < h < n:
            decay[h] = np.nan  # no forward window of this length fits
            continue
        # Compounded t+1..t+h. np.roll wrapped the sample start into the tail,
        # and a plain cumprod let one missing return poison every later window.
        fwd, whole = np.full(n, np.nan), gaps[1 + h:n + 1] == gaps[1:n - h + 1]
        fwd[:n - h] = np.where(whole, cum[1 + h:n + 1] / cum[1:n - h + 1] - 1.0, np.nan)
        valid = ~np.isnan(feature) & ~np.isnan(fwd)
        ic, _ = spearmanr(feature[valid], fwd[valid])
        decay[h] = ic
    return decay  # Expect: decreasing |IC| as h increases

Guardrails

• Non-decaying IC across horizons - strong sign of information leakage
• Always validate on expanding windows - never compute IC on the full sample at once

Production Implementation

from ml4t.diagnostic.api import compute_ic_hac_stats, cross_sectional_ic_series
from ml4t.diagnostic.metrics import analyze_feature_outcome

ic_series = cross_sectional_ic_series(features, forward_returns, pred_col="signal", ret_col="forward_return", entity_col="symbol")
stats = compute_ic_hac_stats(ic_series)  # HAC-corrected t-stats

analysis = analyze_feature_outcome(
    predictions=features, prices=prices, pred_col="prediction",
    price_col="close", date_col="date", group_col="symbol",
)

Checklist

• Every feature has IC computed with p-value < 0.05
• IC stability checked across time (IC-IR > 0.5)
• Any feature with |IC| > 0.10 investigated for leakage
• IC peaks at expected horizon then decays (non-decaying IC suggests leakage)
• Pairwise correlation < 0.7 with all other selected features
• Null percentage < 5% and outliers winsorized