Backtest Integrity and Validation Desk
Overview
The Backtest Integrity skill provides a rigorous framework for auditing, validating, and stress-testing quantitative trading strategies. In quantitative finance and digital asset markets, apparent predictive power is frequently an artifact of data snooping, multiple testing, look-ahead bias, and unrealistic cost assumptions. This skill establishes an epistemological defense against false discoveries, ensuring that trading strategies deployed for operational consideration possess genuine statistical edge rather than phantom alpha generated by flawed backtest engineering.
Rooted in academic consensus—including Bailey and López de Prado on the Deflated Sharpe Ratio, Harvey and Liu on multiple testing haircuts, White's Reality Check, and empirical findings on alpha decay (arXiv 2605.16895)—the skill enforces strict pre-registration protocols, robust null hypothesis testing, realistic transaction cost modeling, and definitive binary kill verdicts.
Primary sources for the claims in this section: https://doi.org/10.3905/jpm.2014.40.5.094 and https://doi.org/10.3905/jpm.2015.42.1.013.
When to use
- Pre-registering hypothesis specifications, parameter grids, and decision rules prior to data execution.
- Issuing definitive PROCEED or KILL verdicts and enforcing re-registration barriers for failed strategies.
When NOT to use
- Collecting raw market data, historical price feeds, or building general data-engineering pipelines (use
ac-market-data-collection).
- Performing forensic post-mortem analysis on historical liquidation crashes or market cascades (use
ac-cascade-forensics).
- Recording individual live trade executions, position sizing ledgers, or discretionary trading journals (use
ac-crypto-trading-discipline).
- Deploying live trading algorithms or managing real-time exchange API connections and order routing infrastructure.
Index of Subskills
| Leaf |
Domain segment |
Scope (one line) |
multiple-testing.md |
crypto-trading/multiple-testing |
How many trials actually happened; deflated Sharpe; Harvey-Liu haircuts; White's Reality Check; the 42-cell/41-window arithmetic as a worked example. |
look-ahead-and-leakage.md |
crypto-trading/look-ahead-and-leakage |
Point-in-time discipline (universe, restatements, feature availability), survivorship, and how a single leaked feature invalidates a backtest. |
sample-and-regime.md |
crypto-trading/sample-and-regime |
Regime heterogeneity (herd vs arbitrage regimes with identical distributions), sample size vs effect size, the one-year/five-ticker limits, and subcriticality's implication (no self-sustaining cascade to simulate). |
cost-and-execution.md |
crypto-trading/cost-and-execution |
Commission, spread, impact, funding, and the Sharpe 0.43→0.22 example; fill assumptions that flatter a backtest. |
placebo-and-null.md |
crypto-trading/placebo-and-null |
Shifted placebos, permutation and bootstrap nulls, the collider problem (sample conditioned on its own outcome), and what a null must be able to reject. |
pre-registration.md |
crypto-trading/pre-registration |
Fixing the analysis before the data: hypothesis, window, cells, test statistic, decision rule; the 42-cell/Bonferroni example; versioning the pre-registration. |
verdict-and-kill.md |
crypto-trading/verdict-and-kill |
The decision rule that kills a strategy, documenting the kill, and the conditions under which a killed strategy may be revisited (new data, new pre-registration — never a re-run on the same sample). |
Status and Honest Limits
This skill operates under strict epistemic boundaries. It evaluates quantitative strategies based on published statistical methods, econometric standards, and empirical findings. It does not predict future market returns, guarantee trading profitability, or endorse specific financial assets. All backtest validations are probabilistic assessments subject to model risk, parameter instability, and structural regime shifts.
Sources
1---2name: ac-backtest-integrity3description: "Auditing whether a trading backtest can be believed: multiple-testing deflation, look-ahead and survivorship leakage, sample and regime adequacy, execution-cost honesty, placebo and null tests, pre-registration, and the decision rule that kills a strategy. Audits evidence; never recommends a strategy. Use when asking \"is this backtest real\", \"validate my strategy\", \"deflated Sharpe\", \"look-ahead bias\", \"pre-register a study\". Not for data collection (ac-market-data-collection), forensics (ac-cascade-forensics), or the trade record (ac-crypto-trading-discipline)."4license: MIT5metadata:6 source: https://skillmd.com/skills/jiayaoqijia/ac-backtest-integrity7 domain: crypto-trading8 status: seedling9---1011# Backtest Integrity and Validation Desk1213## Overview1415The Backtest Integrity skill provides a rigorous framework for auditing, validating, and stress-testing quantitative trading strategies. In quantitative finance and digital asset markets, apparent predictive power is frequently an artifact of data snooping, multiple testing, look-ahead bias, and unrealistic cost assumptions. This skill establishes an epistemological defense against false discoveries, ensuring that trading strategies deployed for operational consideration possess genuine statistical edge rather than phantom alpha generated by flawed backtest engineering. 1617Rooted in academic consensus—including Bailey and López de Prado on the Deflated Sharpe Ratio, Harvey and Liu on multiple testing haircuts, White's Reality Check, and empirical findings on alpha decay (arXiv 2605.16895)—the skill enforces strict pre-registration protocols, robust null hypothesis testing, realistic transaction cost modeling, and definitive binary kill verdicts.1819Primary sources for the claims in this section: <https://doi.org/10.3905/jpm.2014.40.5.094> and <https://doi.org/10.3905/jpm.2015.42.1.013>.2021## When to use2223- Pre-registering hypothesis specifications, parameter grids, and decision rules prior to data execution.24- Issuing definitive PROCEED or KILL verdicts and enforcing re-registration barriers for failed strategies.2526## When NOT to use2728- Collecting raw market data, historical price feeds, or building general data-engineering pipelines (use `ac-market-data-collection`).29- Performing forensic post-mortem analysis on historical liquidation crashes or market cascades (use `ac-cascade-forensics`).30- Recording individual live trade executions, position sizing ledgers, or discretionary trading journals (use `ac-crypto-trading-discipline`).31- Deploying live trading algorithms or managing real-time exchange API connections and order routing infrastructure.3233## Index of Subskills3435| Leaf | Domain segment | Scope (one line) |36| :--- | :--- | :--- |37| [`multiple-testing.md`](subskills/multiple-testing.md) | `crypto-trading/multiple-testing` | How many trials actually happened; deflated Sharpe; Harvey-Liu haircuts; White's Reality Check; the 42-cell/41-window arithmetic as a worked example. |38| [`look-ahead-and-leakage.md`](subskills/look-ahead-and-leakage.md) | `crypto-trading/look-ahead-and-leakage` | Point-in-time discipline (universe, restatements, feature availability), survivorship, and how a single leaked feature invalidates a backtest. |39| [`sample-and-regime.md`](subskills/sample-and-regime.md) | `crypto-trading/sample-and-regime` | Regime heterogeneity (herd vs arbitrage regimes with identical distributions), sample size vs effect size, the one-year/five-ticker limits, and subcriticality's implication (no self-sustaining cascade to simulate). |40| [`cost-and-execution.md`](subskills/cost-and-execution.md) | `crypto-trading/cost-and-execution` | Commission, spread, impact, funding, and the Sharpe 0.43→0.22 example; fill assumptions that flatter a backtest. |41| [`placebo-and-null.md`](subskills/placebo-and-null.md) | `crypto-trading/placebo-and-null` | Shifted placebos, permutation and bootstrap nulls, the collider problem (sample conditioned on its own outcome), and what a null must be able to reject. |42| [`pre-registration.md`](subskills/pre-registration.md) | `crypto-trading/pre-registration` | Fixing the analysis before the data: hypothesis, window, cells, test statistic, decision rule; the 42-cell/Bonferroni example; versioning the pre-registration. |43| [`verdict-and-kill.md`](subskills/verdict-and-kill.md) | `crypto-trading/verdict-and-kill` | The decision rule that kills a strategy, documenting the kill, and the conditions under which a killed strategy may be revisited (new data, new pre-registration — never a re-run on the same sample). |4445## Status and Honest Limits4647This skill operates under strict epistemic boundaries. It evaluates quantitative strategies based on published statistical methods, econometric standards, and empirical findings. It does not predict future market returns, guarantee trading profitability, or endorse specific financial assets. All backtest validations are probabilistic assessments subject to model risk, parameter instability, and structural regime shifts.4849## Sources5051- Bailey, D. H., & López de Prado, M. (2014). The Deflated Sharpe Ratio. Journal of Portfolio Management. https://doi.org/10.3905/jpm.2014.40.5.09452- Harvey, C. R., & Liu, Y. (2015). Backtesting. Journal of Portfolio Management. https://doi.org/10.3905/jpm.2015.42.1.01353- White, H. (2000). A Reality Check for Data Snooping. Econometrica. https://doi.org/10.1111/1468-0262.0015254- arXiv:2605.16895. Alpha Illusion. https://arxiv.org/abs/2605.1689555- Open Science Framework (OSF). Pre-Registration Standards. https://osf.io/prereg/