ml4t Factor Evaluation
A factor that looks predictive may be untradeable due to high turnover, rapid decay, or non-monotonic quantile spreads. Comprehensive evaluation before portfolio integration prevents costly live failures.
The Problem
Reporting a single backtest Sharpe ratio conflates signal quality with portfolio construction. A factor with IC 0.03 and low turnover can be more valuable than one with IC 0.05 and 80% daily turnover - whether IC is sufficient depends on breadth, turnover costs, and regime stability. Without decomposing signal quality into IC, quantile monotonicity, turnover, and decay, you cannot diagnose why a strategy fails or how to improve it.
The Pattern
WRONG
# Evaluate only via backtest Sharpe - hides factor-level issues
returns = run_backtest(signal)
sharpe = returns.mean() / returns.std() * np.sqrt(252)
print(f"Sharpe: {sharpe:.2f}") # No idea why it works or doesn't
CORRECT
import numpy as np
from scipy import stats
# 1. Information Coefficient: rank correlation with forward returns
def compute_ic_series(signal, forward_returns, timestamps):
"""Per-period rank IC between signal and forward returns."""
ic_by_period = []
for t in np.unique(timestamps):
mask = timestamps == t
if mask.sum() < 10:
continue
ic, _ = stats.spearmanr(signal[mask], forward_returns[mask])
ic_by_period.append(ic)
return np.array(ic_by_period)
ic_series = compute_ic_series(signal, fwd_ret, dates)
print(f"IC Mean: {np.mean(ic_series):.4f}")
print(f"IC Std: {np.std(ic_series):.4f}")
print(f"IC IR: {np.mean(ic_series) / np.std(ic_series):.3f}")
print(f"IC t-stat: {np.mean(ic_series) / (np.std(ic_series) / np.sqrt(len(ic_series))):.2f}")
# 2. Quantile spreads: returns by signal quintile
n_quantiles = 5
quantile_labels = np.ceil(stats.rankdata(signal) / len(signal) * n_quantiles)
for q in range(1, n_quantiles + 1):
q_ret = fwd_ret[quantile_labels == q].mean()
print(f" Q{q}: {q_ret:.4f}")
# Good factor: monotonic Q1 < Q2 < ... < Q5 (or reverse)
# 3. Turnover: how often positions change
quantiles_today = np.ceil(stats.rankdata(signal_today) / len(signal_today) * 5)
quantiles_yesterday = np.ceil(stats.rankdata(signal_yesterday) / len(signal_yesterday) * 5)
turnover = (quantiles_today != quantiles_yesterday).mean()
print(f"Turnover: {turnover:.1%}")
IC Decay Analysis
Signal power fades over time. Measure IC at multiple horizons to find optimal rebalance frequency:
for horizon in [1, 5, 10, 21, 63]:
fwd = returns.shift(-horizon)
ic = stats.spearmanr(signal[~np.isnan(fwd)], fwd[~np.isnan(fwd)])[0]
print(f" {horizon:>3}d IC: {ic:.4f}")
# IC peaks at the natural signal horizon; decays beyond it
Guardrails
- IC alone is insufficient - a factor with high IC but non-monotonic quintiles is unreliable
- High turnover (>30% daily) signals that transaction costs may consume the alpha
- Always compute t-statistics with HAC (Newey-West) standard errors for autocorrelated IC series
- Decay analysis determines rebalance frequency - rebalancing faster than the peak-IC horizon wastes costs
- IC is a learnability screen, not a strategy - a positive IC only means the signal has information; tradability requires checking turnover, costs, and capacity
Production Implementation
ml4t-diagnostic provides validated IC computation with HAC corrections:
from ml4t.diagnostic.api import compute_ic_hac_stats, cross_sectional_ic_series
ic_series = cross_sectional_ic_series(
signal_frame,
return_frame,
pred_col="signal",
ret_col="forward_return",
date_col="date",
entity_col="symbol",
)
ic_stats = compute_ic_hac_stats(ic_series) # Newey-West corrected
print(f"IC: {ic_stats['mean_ic']:.4f} (t={ic_stats['t_stat']:.2f})")
Checklist
- IC mean, IC IR, and worst-fold IC computed (IC IR > 0.5; worst-fold IC same sign as mean)
- Quantile returns checked for monotonicity
- Turnover estimated and compared against cost model
- Decay analysis performed to determine rebalance frequency
- t-statistics use HAC standard errors, not naive SE