Backtest Overfitting

Decide whether an edge that passed every mechanical check is still just the best of N tries. TRIGGER - PBO, probability of backtest overfitting, CSCV, combinatorially symmetric cross validation, logit of the out-of-sample rank; minimum backtest length, MinBTL, "how much history do I need", "how many parameter sets can I try on N years of data"; a grid search, Optuna or AutoML picked a winner and it decayed; "in-sample Sharpe 2, live Sharpe 0", "the best parameter set stopped working", "is this peak on my parameter surface real"; pypbo, RiskLabAI CSCV. SKIP for the deflated and probabilistic Sharpe ratios, the trial ledger and SPA/StepM/MCS against a benchmark (backtest-validation); for family-wise error and false-discovery control over a whole research programme (multiple-testing-ledger); for the purged and combinatorial CV splitters themselves (lib-purgedcv); for the mechanical leakage gates that come first (research-integrity-guards); and for regime coverage of the test window (regime-detection).

howard-lynn-ye Updated 1 repo stars

File contents

howard-lynn-ye/fin-skills commit 3e6ee4f456

Frequently asked questions

npx get-skill add howard-lynn-ye/backtest-overfitting