Ml4t Latent Factors

Latent Factor Extraction

PCA explains 80% of return variance - but variance is not alpha. The first principal component captures market beta, which earns the equity premium, not a tradeable edge. Confusing variance-explained with pricing power is the central mistake.

The Problem

With 400+ published return predictors, hand-picking factors invites overfitting. Latent factor methods (PCA, autoencoders) extract structure directly from data. But three failure modes undermine them:

1. Variance != pricing - high-variance factors may capture idiosyncratic noise, not compensated risk.
2. Eigenvector instability - when assets (N) approach time periods (T), sample covariance is dominated by noise. Marchenko-Pastur theory gives the noise boundary.
3. Full-sample PCA is leakage - fitting PCA on the complete panel, then testing on a held-out period, leaks the covariance structure of the test period into training.

The Pattern

WRONG

from sklearn.decomposition import PCA
import numpy as np

# Fit PCA on FULL return panel, then use factors for prediction
pca = PCA(n_components=5)
factors = pca.fit_transform(returns_panel)  # Full-sample fit = leakage

# "80% variance explained" - but does it predict returns?
print(f"Explained variance: {pca.explained_variance_ratio_.sum():.1%}")
signal = factors[:, 0]  # Assumes first PC predicts returns

CORRECT

from sklearn.decomposition import PCA
import numpy as np

def mp_upper_bound(n_assets, n_periods):
    """Marchenko-Pastur noise threshold for a correlation matrix."""
    gamma = n_assets / n_periods
    return (1 + np.sqrt(gamma)) ** 2

# Walk-forward PCA: fit on training window only
def walk_forward_pca(returns, n_components=5, train_window=504):  # 504 ≈ 2 years
    """Fit per fold; keep only components above that fold's noise bound."""
    factors = np.full((len(returns), n_components), np.nan)
    n_assets = returns.shape[1]
    for t in range(train_window, len(returns)):
        train = returns[t - train_window:t]
        # Standardise on the training fold: the bound is stated for a
        # correlation matrix, so raw return variances are not comparable.
        mu, sd = train.mean(0), train.std(0)
        pca = PCA(n_components=n_components)
        pca.fit((train - mu) / sd)
        scores = pca.transform((returns[t:t + 1] - mu) / sd)[0]
        # Drop this fold's noise components here. Averaging eigenvalues over
        # every fold and filtering once would select using future data.
        keep = pca.explained_variance_ > mp_upper_bound(n_assets, train_window)
        factors[t] = np.where(keep, scores, np.nan)
    return factors

factors = walk_forward_pca(returns)
kept = np.mean(np.any(~np.isnan(factors), axis=0))
print(f"Components above the noise bound in at least one fold: {kept:.0%}")

Method Comparison

Start with PCA + Marchenko-Pastur filtering. Graduate to IPCA only if characteristics drive time-varying exposures.

Guardrails

• Walk-forward fit: PCA must be re-fit on each fold's training data - never on the full panel
• Marchenko-Pastur test: discard components below the random-matrix noise bound
• Microcap bias: equal-weighted PCA is dominated by small/illiquid stocks - use market-cap weighting or NYSE breakpoints
• Loading rotation: eigenvectors are not stable across subperiods - don't assign fixed economic labels ("this is momentum")
• Autoencoder seeds: report results across 10+ random seeds; single-seed results are unreliable

Production Implementation

ml4t-diagnostic provides factor evaluation infrastructure for custom latent factor outputs:

from ml4t.diagnostic.api import cross_sectional_ic_series

# After walk-forward PCA, evaluate factor predictiveness via IC
ic = cross_sectional_ic_series(
    predictions=factor_df,      # DataFrame with date, symbol, prediction
    returns=forward_returns_df,  # DataFrame with date, symbol, forward_return
    date_col="date",
    entity_col="symbol",
)

Checklist

• PCA fit walk-forward per fold, not on full panel
• Marchenko-Pastur noise threshold applied to discard noise components
• Variance-explained distinguished from predictive power (IC tested)
• Microcap bias controlled (cap-weighted or filtered universe)
• Results stable across random seeds (for autoencoders)