ml4t Case Study Development Workflow
A case study is not just a pipeline; it is a sequence of research decisions. Clean artifacts are necessary, but the real question is whether each stage earned the right to move to the next one.
The Problem
A researcher puts data loading, feature engineering, model training, and backtesting in a single notebook. It runs once and produces a Sharpe ratio. Later, they change the label horizon and must re-run everything. The model section silently reads stale features. The backtest uses predictions from a previous run. Nobody knows which config produced the final result. The pipeline is unreproducible and the results are untrustworthy.
The Pattern
WRONG
import lightgbm as lgb
import polars as pl
# Everything in one notebook - no artifact boundaries, no config
data = pl.read_parquet("etf_data.parquet")
features = data.select(["momentum_12m", "volatility_20d"])
labels = data["close"].pct_change(21).shift(-21)
model = lgb.LGBMRegressor().fit(features, labels) # Full-sample train!
predictions = model.predict(features)
# Backtest on training predictions with no cost model
cumulative_return = (predictions * labels).cumsum()
print(f"Sharpe: {sharpe(cumulative_return):.2f}") # Meaningless number
CORRECT
# Seven-stage pipeline - each stage reads upstream artifacts, writes downstream
# All config lives in setup.yaml (SSOT), not scattered across notebooks
# Stage 1: SETUP (Ch6) - define universe, frequency, costs, evaluation protocol
# → Document hypothesis, kill criteria, and capacity estimate in setup.yaml
# → Output: setup.yaml with universe, frequency, cost_model, cv_config
# Stage 2: LABELS (Ch7) - forward returns and/or triple-barrier labels
# → Use ml4t-triple-barrier skill for event-driven labels
# → Output: data/labels/fwd_ret_{horizon}.parquet
# Stage 3: FEATURES (Ch8) - financial and model-based features
# → Use ml4t-compute-features skill
# → Output: data/features/financial.parquet, data/features/model_based.parquet
# Stage 4: EVALUATE FEATURES (Ch7-8) - IC, stability, redundancy
# → Use ml4t-evaluate-factor skill for IC significance and decay
# → Gate: drop features with IC t-stat < 2.0 or sign flip across subperiods
# Stage 5: TRAIN MODELS (Ch11+) - across CV folds, register each run
# → Use ml4t-cpcv skill for combinatorial purged cross-validation
# → Output: run_log/models/{hash}/predictions.parquet per config
# → Register: training_run → prediction_set entries in registry
# Stage 6: BACKTEST (Ch16) - combine predictions, simulate trading
# → Use ml4t-run-backtest skill with realistic cost model
# → Output: run_log/strategy/{hash}/ with equity curve, metrics
# Stage 7: SYNTHESIZE (Ch20) - compare models, report best config
# → Output: results/*.json with provenance back to registry
Stage Gates
| Gate | Core question | Advance only if |
|---|---|---|
| Setup | Is the hypothesis falsifiable? | Universe, costs, and success criteria are explicit |
| Labels | Is the target economically meaningful? | Horizon and label logic match holding period |
| Features | Do the signals survive diagnostics? | IC is stable, joins are clean, leakage checks pass |
| Models | Is validation credible? | CPCV or holdout beats a baseline without leakage |
| Backtest | Does the economics survive friction? | Costs, turnover, and capacity remain acceptable |
| Synthesis | Is there a decision? | Best config and rejection reasons are documented |
Artifact contracts still matter; use ml4t-case-study-pipeline to define them. This skill governs whether the study should advance.
Guardrails
- If the hypothesis changes materially, restart at Setup instead of patching downstream results
- If model training runs without a registry entry, downstream conclusions are not auditable
- If backtest Sharpe > 2.0 on daily data, suspect leakage before celebrating
- If a gate fails, record the rejection reason instead of quietly continuing
Production Implementation
ml4t-engineer and ml4t-backtest cover the feature and simulation handoff:
from ml4t.backtest import BacktestConfig, run_backtest
from ml4t.engineer import compute_features
features = compute_features(data, "configs/features.yaml")
backtest_config = BacktestConfig.from_yaml("configs/backtest.yaml")
results = run_backtest(prices=prices, signals=signals, strategy=strategy, config=backtest_config)
Checklist
- Setup gate defines universe, horizon, costs, and success criteria
- Feature gate passes IC, stability, and leakage checks before modeling
- Model gate beats a baseline on CPCV or holdout, not one lucky split
- Backtest gate survives realistic costs and turnover constraints
- Final synthesis records the chosen configuration and rejected alternatives