# Ml4t Feature Validation

> Validate features before training - IC significance, stability, redundancy, and contamination checks. Use when auditing feature quality before model fitting.

- Skill: `ml4t/ml4t-feature-validation` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ml4t/ml4t-feature-validation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ml4t/ml4t-feature-validation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: ml4t (https://skillmd.com/u/ml4t)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ml4t/ml4t-feature-validation

---

# Feature Validation

A feature with IC of 0.05 on the full sample may have IC of 0.12 in one year and -0.03 in every other year. Without validation, the model trains on noise disguised as signal.

## The Problem

Skipping feature validation leads to three failures: lookahead contamination,
regime-specific features that fail live, and redundant features that waste model capacity.

## The Pattern

### WRONG
```python
from sklearn.ensemble import GradientBoostingRegressor

# Train on all features without any validation - overfitting guaranteed
model = GradientBoostingRegressor(n_estimators=200)
model.fit(X_train, y_train)  # 50 features, no idea which are noise
```

### CORRECT
```python
from scipy.stats import spearmanr
import numpy as np

def validate_feature(feature: np.ndarray, target: np.ndarray, dates: np.ndarray) -> dict:
    """Screen a single feature for predictive quality."""
    ok = ~np.isnan(feature) & ~np.isnan(target)  # both: one nan makes ic nan
    ic, p_value = spearmanr(feature[ok], target[ok])
    quarters = dates.astype("datetime64[M]").astype(int) // 3  # numpy has no [Q]
    quarterly_ics = []
    for q in np.unique(quarters):
        mask = (quarters == q) & ok  # ok too: the floor counts VALID pairs
        if mask.sum() > 30:
            qic, _ = spearmanr(feature[mask], target[mask])
            quarterly_ics.append(qic)

    quarterly_ics = np.array(quarterly_ics)
    quarterly_ics = quarterly_ics[~np.isnan(quarterly_ics)]  # nan != a bad quarter
    ic_mean, ic_std = quarterly_ics.mean(), quarterly_ics.std()
    ic_ir = ic_mean / ic_std if ic_std > 0 else 0  # IC information ratio
    leakage_flag = abs(ic) > 0.10

    return {"ic": ic, "p_value": p_value, "ic_ir": ic_ir, "leakage_flag": leakage_flag,
            "pct_positive_quarters": np.mean(quarterly_ics > 0)}
```

## Validation Checklist Sequence

| Step | Check | Pass Criteria |
|------|-------|---------------|
| 1. Completeness | Null percentage | < 5% (or documented imputation) |
| 2. Outliers | Values beyond 5 sigma | < 1% (winsorize if needed) |
| 3. IC significance | Spearman rank correlation | p-value < 0.05 |
| 4. IC stability | Quarterly IC information ratio | IC-IR > 0.5 |
| 5. Leakage screen | Absolute IC threshold | \|IC\| < 0.10 or explained mechanism |
| 6. Redundancy | Pairwise correlation with existing features | < 0.7 |

## IC Decay Analysis

```python
def ic_decay(feature: np.ndarray, returns: np.ndarray, horizons: list[int]) -> dict:
    """IC should decay with horizon - if it doesn't, suspect leakage."""
    n, decay = len(returns), {}
    gaps = np.r_[0, np.cumsum(np.isnan(returns))]  # missing returns so far
    cum = np.r_[1.0, np.cumprod(1.0 + np.nan_to_num(returns))]
    for h in horizons:
        if not 0 < h < n:
            decay[h] = np.nan  # no forward window of this length fits
            continue
        # Compounded t+1..t+h. np.roll wrapped the sample start into the tail,
        # and a plain cumprod let one missing return poison every later window.
        fwd, whole = np.full(n, np.nan), gaps[1 + h:n + 1] == gaps[1:n - h + 1]
        fwd[:n - h] = np.where(whole, cum[1 + h:n + 1] / cum[1:n - h + 1] - 1.0, np.nan)
        valid = ~np.isnan(feature) & ~np.isnan(fwd)
        ic, _ = spearmanr(feature[valid], fwd[valid])
        decay[h] = ic
    return decay  # Expect: decreasing |IC| as h increases
```

## Guardrails

- **Non-decaying IC across horizons** - strong sign of information leakage
- **Always validate on expanding windows** - never compute IC on the full sample at once

## Production Implementation

```python
from ml4t.diagnostic.api import compute_ic_hac_stats, cross_sectional_ic_series
from ml4t.diagnostic.metrics import analyze_feature_outcome

ic_series = cross_sectional_ic_series(features, forward_returns, pred_col="signal", ret_col="forward_return", entity_col="symbol")
stats = compute_ic_hac_stats(ic_series)  # HAC-corrected t-stats

analysis = analyze_feature_outcome(
    predictions=features, prices=prices, pred_col="prediction",
    price_col="close", date_col="date", group_col="symbol",
)
```

## Checklist

- [ ] Every feature has IC computed with p-value < 0.05
- [ ] IC stability checked across time (IC-IR > 0.5)
- [ ] Any feature with |IC| > 0.10 investigated for leakage
- [ ] IC peaks at expected horizon then decays (non-decaying IC suggests leakage)
- [ ] Pairwise correlation < 0.7 with all other selected features
- [ ] Null percentage < 5% and outliers winsorized