# Ml4t Latent Factors

> Extract latent factors from return panels using PCA, IPCA, or autoencoders with proper noise diagnostics. Use when reducing dimensionality or discovering risk structure.

- Skill: `ml4t/ml4t-latent-factors` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ml4t/ml4t-latent-factors`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ml4t/ml4t-latent-factors/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: ml4t (https://skillmd.com/u/ml4t)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ml4t/ml4t-latent-factors

---

# Latent Factor Extraction

PCA explains 80% of return variance - but variance is not alpha. The first principal component captures market beta, which earns the equity premium, not a tradeable edge. Confusing variance-explained with pricing power is the central mistake.

## The Problem

With 400+ published return predictors, hand-picking factors invites overfitting. Latent factor methods (PCA, autoencoders) extract structure directly from data. But three failure modes undermine them:

1. **Variance != pricing** - high-variance factors may capture idiosyncratic noise, not compensated risk.
2. **Eigenvector instability** - when assets (N) approach time periods (T), sample covariance is dominated by noise. Marchenko-Pastur theory gives the noise boundary.
3. **Full-sample PCA is leakage** - fitting PCA on the complete panel, then testing on a held-out period, leaks the covariance structure of the test period into training.

## The Pattern

### WRONG
```python
from sklearn.decomposition import PCA
import numpy as np

# Fit PCA on FULL return panel, then use factors for prediction
pca = PCA(n_components=5)
factors = pca.fit_transform(returns_panel)  # Full-sample fit = leakage

# "80% variance explained" - but does it predict returns?
print(f"Explained variance: {pca.explained_variance_ratio_.sum():.1%}")
signal = factors[:, 0]  # Assumes first PC predicts returns
```

### CORRECT
```python
from sklearn.decomposition import PCA
import numpy as np

def mp_upper_bound(n_assets, n_periods):
    """Marchenko-Pastur noise threshold for a correlation matrix."""
    gamma = n_assets / n_periods
    return (1 + np.sqrt(gamma)) ** 2

# Walk-forward PCA: fit on training window only
def walk_forward_pca(returns, n_components=5, train_window=504):  # 504 ≈ 2 years
    """Fit per fold; keep only components above that fold's noise bound."""
    factors = np.full((len(returns), n_components), np.nan)
    n_assets = returns.shape[1]
    for t in range(train_window, len(returns)):
        train = returns[t - train_window:t]
        # Standardise on the training fold: the bound is stated for a
        # correlation matrix, so raw return variances are not comparable.
        mu, sd = train.mean(0), train.std(0)
        pca = PCA(n_components=n_components)
        pca.fit((train - mu) / sd)
        scores = pca.transform((returns[t:t + 1] - mu) / sd)[0]
        # Drop this fold's noise components here. Averaging eigenvalues over
        # every fold and filtering once would select using future data.
        keep = pca.explained_variance_ > mp_upper_bound(n_assets, train_window)
        factors[t] = np.where(keep, scores, np.nan)
    return factors

factors = walk_forward_pca(returns)
kept = np.mean(np.any(~np.isnan(factors), axis=0))
print(f"Components above the noise bound in at least one fold: {kept:.0%}")
```

## Method Comparison

| Method | Strengths | Limitations |
|--------|-----------|-------------|
| PCA | Fast, linear, interpretable loadings | Static betas, variance != pricing |
| IPCA | Dynamic betas via characteristics | Sensitive to characteristic selection |
| Autoencoder | Non-linear factor structure | Overfits without adversarial constraints |
| RP-PCA | Incorporates risk-premium signal | Requires pricing-error objective |

Start with PCA + Marchenko-Pastur filtering. Graduate to IPCA only if characteristics drive time-varying exposures.

## Guardrails

- **Walk-forward fit**: PCA must be re-fit on each fold's training data - never on the full panel
- **Marchenko-Pastur test**: discard components below the random-matrix noise bound
- **Microcap bias**: equal-weighted PCA is dominated by small/illiquid stocks - use market-cap weighting or NYSE breakpoints
- **Loading rotation**: eigenvectors are not stable across subperiods - don't assign fixed economic labels ("this is momentum")
- **Autoencoder seeds**: report results across 10+ random seeds; single-seed results are unreliable

## Production Implementation

`ml4t-diagnostic` provides factor evaluation infrastructure for custom latent factor outputs:

```python
from ml4t.diagnostic.api import cross_sectional_ic_series

# After walk-forward PCA, evaluate factor predictiveness via IC
ic = cross_sectional_ic_series(
    predictions=factor_df,      # DataFrame with date, symbol, prediction
    returns=forward_returns_df,  # DataFrame with date, symbol, forward_return
    date_col="date",
    entity_col="symbol",
)
```

## Checklist

- [ ] PCA fit walk-forward per fold, not on full panel
- [ ] Marchenko-Pastur noise threshold applied to discard noise components
- [ ] Variance-explained distinguished from predictive power (IC tested)
- [ ] Microcap bias controlled (cap-weighted or filtered universe)
- [ ] Results stable across random seeds (for autoencoders)