# Trading Strategy Review > Review trading strategies and backtests for the failure modes that blow up live — look-ahead bias, overfitting, survivorship bias, unrealistic fills, and risk-of-ruin. Grounded in backtest-overfitting research (PBO, Deflated Sharpe). Outputs severity-rated findings with fixes. Use to vet a strategy, audit a backtest, or pre-deployment risk-check. - Skill: `viprasol-tech/trading-strategy-review` (Agent Skill, multi-file: 6 files) - Install (CLI): `npx skillmds@latest add viprasol-tech/trading-strategy-review` - Raw SKILL.md: https://api.skillmd.com/api/skills/viprasol-tech/trading-strategy-review/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Security - Author: Viprasol-Tech (https://skillmd.com/u/viprasol-tech) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/viprasol-tech/trading-strategy-review --- # Trading Strategy & Backtest Review A methodology for reviewing a trading strategy, a backtest, or strategy code (Python, Pine, or any language) for the failure modes that make a backtest look great in a chart and lose money in production. The output is a **severity-rated findings report** with a concrete fix for each problem. This skill is grounded in the established backtest-overfitting literature, primarily the work of **David H. Bailey and Marcos López de Prado** — *"The Probability of Backtest Overfitting"* (PBO), *"The Deflated Sharpe Ratio"*, *"Pseudo-Mathematics and Financial Charlatanism"*, and *"The Seven Sins of Quantitative Investing."* The central insight of that literature: if you try enough strategy configurations, **a spectacular in-sample backtest is the expected outcome even when the strategy has zero real edge.** Most of the checklist below exists to detect exactly that. > **Not financial advice.** This skill performs an *educational, methodological review of > a strategy or backtest*. It is not investment advice, not a recommendation to trade, and > not a guarantee of any outcome. Past backtest performance — even after every fix here — > does **not** guarantee live results. The user is solely responsible for their own > trading and risk decisions. --- ## When to Activate Activate this skill when the user: - Shares a trading strategy idea, ruleset, or signal and asks whether it is sound. - Shares **backtest code** (Python/pandas/`backtrader`/`vectorbt`/`zipline`, Pine Script, or any language) and asks for a review. - Shares **backtest results** — an equity curve, a metrics table (Sharpe, CAGR, max drawdown, win rate), or a screenshot from TradingView / a backtesting platform. - Asks for a "pre-deployment check," "is this overfit?", "why might this fail live?", or a risk review before putting real capital behind a strategy. If the user only shares *results* with no code, you can still run Steps 1, 2, 4 and 5 and the metrics sanity check; flag that code-level confirmation is needed for several findings. --- ## Step 1: Scope & Context Before reviewing, establish context. Ask for anything missing — these answers change the severity of nearly every finding. If the user cannot answer, note the gap as an `Info` finding and assume the worst reasonable case. | Question | Why it matters | |---|---| | **Asset class & instrument** (equities, futures, FX, crypto, options) | Drives cost model, liquidity, borrow/funding, session boundaries. | | **Timeframe / bar size** (tick, 1m, 1h, daily) | Sets the realistic Sharpe ceiling and how dangerous look-ahead is. | | **Backtest window & in-sample period** | Short windows + one regime = fragile. | | **Number of parameters** in the strategy | More knobs → more overfitting capacity. | | **Number of strategies / configs tried** (the "trials") | The single biggest driver of false discovery; needed for Deflated Sharpe / PBO. | | **Out-of-sample / walk-forward done?** | If everything is in-sample, results are unverified. | | **Live or paper track record?** | Live > paper > backtest in evidentiary weight. | | **Capital, position size, leverage** | Determines risk-of-ruin and market-impact realism. | | **Costs modeled?** (commission, slippage, spread, borrow, funding) | Frictionless backtests are the most common silent killer. | --- ## Step 2: Severity Model Rate every finding with this scale. The driving question is always: *"Does this change the decision to trade, or just the precision of the estimate?"* | Severity | Definition | Examples | |---|---|---| | **Critical** | Invalidates the results entirely, or exposes the account to ruin. The backtest cannot be trusted and/or the strategy can blow up. | Look-ahead bias in the signal; zero transaction costs on a high-turnover strategy; no risk limit / unbounded position sizing; survivorship-biased universe; results are 100% in-sample with hundreds of trials. | | **High** | Materially inflates performance or materially understates risk; likely flips the strategy from "edge" to "no edge" once corrected. | Optimistic fills (mid/close, no slippage); over-parameterization without OOS; ignoring borrow/funding; regime dependence on a single bull market. | | **Medium** | Biases results in a meaningful but recoverable way; needs correction before sizing real money. | Curve-fit stop/target levels; survivorship in a secondary filter; small trade count weakening significance; timezone/session edge cases. | | **Low** | Minor realism or robustness gap; unlikely to flip the conclusion but should be fixed. | Rounding/contract-size handling; minor parameter sensitivity; cost estimate slightly optimistic. | | **Info** | Context, missing information, or good practice to confirm — not a defect per se. | "Number of trials not disclosed"; "confirm data is point-in-time"; "consider regime-tagging the equity curve." | A single **Critical** finding means *do not deploy* until fixed, regardless of how good the metrics look. --- ## Step 3: Failure-Mode Checklist Walk every item. For each, you have: **what it is → how to detect it → why it inflates performance → the fix.** Cover at minimum the 16 modes below. ### 3.1 Look-ahead bias *(usually Critical)* - **What:** The strategy uses information at bar *t* that was not actually available until *t+1* or later — future prices, end-of-bar values used intrabar, same-bar close used to decide a same-bar entry, full-series statistics (mean, z-score, scaler) computed over data that includes the future. - **Detect:** Signals that index the future (`close[t+1]`, `df['close'].shift(-1)`, `.shift(-n)` feeding a feature); indicators computed on the *entire* series then used at earlier bars; `fillna(method='bfill')`; a `StandardScaler`/`MinMaxScaler` fit on the whole dataset before the train/test split; using a bar's high/low to decide entries that also assume the favorable extreme was filled; resampling that leaks the closing bar. - **Why it inflates:** Trading with tomorrow's information is the most powerful "edge" there is — and it is entirely fake. It typically produces unrealistically smooth equity curves and Sharpe ratios that are physically impossible. - **Fix:** Use **point-in-time data** and a **frozen, as-of view** of every input. Lag all signals by one bar relative to execution. Fit any normalizer/scaler **inside** the walk-forward fold on training data only. Decide on bar *t*'s **closed** values, execute at bar *t+1*'s open (or next available price). ```python # BAD — decides and fills on the same bar's close (look-ahead) signal = (df['close'] > df['sma']).astype(int) df['ret'] = signal * df['close'].pct_change() # uses same-bar close # GOOD — decide on closed bar t, execute next bar signal = (df['close'] > df['sma']).astype(int) df['position'] = signal.shift(1) # act on prior bar's signal df['ret'] = df['position'] * df['close'].pct_change() ``` ### 3.2 Survivorship bias *(High–Critical)* - **What:** The universe only contains instruments that survived to today — delisted, bankrupt, merged, or relegated names are missing. - **Detect:** Universe pulled from a *current* index membership / current ticker list; equities backtest with no delisting returns; crypto backtest excluding dead coins; using "today's S&P 500" over a 20-year history. - **Why it inflates:** You systematically exclude the losers. Buy-and-hold and especially mean-reversion strategies look far better than reality because the names that went to zero were never in the test. - **Fix:** Use a **point-in-time, survivorship-free universe** with historical constituents and delisting/bankruptcy returns. Reconstruct index membership as of each date. ### 3.3 Data-snooping / overfitting (too many params, in-sample tuning) *(Critical)* - **What:** Parameters were chosen by maximizing performance on the same data used to report it; or the strategy has so many free parameters it can fit noise. - **Detect:** Grid/optimizer over many parameters reporting only the *best* result; no held-out data; parameter count high relative to trade count; "we tried a few and kept the best." - **Why it inflates:** With enough knobs and enough trials, you can fit historical noise perfectly. Per Bailey–López de Prado, the **expected maximum** in-sample Sharpe across N independent trials grows roughly like √(2·ln N) even when true Sharpe is zero — so a great backtest is the *default* outcome of a search, not evidence of edge. - **Fix:** **Out-of-sample holdout + walk-forward** analysis. Minimize parameters (parsimony). Report the **Deflated Sharpe Ratio** and **PBO** (below). Pre-register the rule before testing where possible. ### 3.4 Multiple testing / p-hacking *(Critical)* - **What:** Many configurations, indicators, or universes were tried; only the winner is reported. The reported Sharpe is a **maximum over trials**, not a single draw. - **Detect:** "Selected the best of K"; many indicators/lookbacks scanned; the count of trials is undisclosed or large. - **Why it inflates:** The more you try, the higher the best in-sample result purely by chance. A nominal Sharpe of 2.0 over 1,000 trials may be statistically indistinguishable from luck. - **Fix:** **Deflated Sharpe Ratio (DSR)** — adjust the observed Sharpe for the number of trials, the variance of trial Sharpes, and the non-normality (skew/kurtosis) of returns; DSR is the probability the true Sharpe > 0 after that haircut. **Probability of Backtest Overfitting (PBO)** via Combinatorially Symmetric Cross-Validation (CSCV): the fraction of splits where the in-sample-best config underperforms the median out-of-sample. **PBO > 0.5 means the selection process is overfit.** Always report **how many trials** were run. ### 3.5 Unrealistic fills *(High–Critical)* - **What:** Orders assumed to fill at prices you could not actually get — mid, close, best-case high/low of the bar, or instantly with no queue. - **Detect:** Backtest fills at the signal bar's `close`; limit orders assumed filled whenever price merely *touched* the level; market-on-open with no gap handling; no spread. - **Why it inflates:** Best-case fills quietly add returns every single trade; the effect is largest for high-frequency / high-turnover strategies. - **Fix:** Model the **bid/ask spread**; fill market orders at the **next bar's open** (or worse); for limit orders, require the price to **trade through** the level, not just touch it, and account for queue position. Add conservative slippage (below). ### 3.6 Slippage & commission omitted/under-modeled *(High–Critical)* - **What:** No (or token) per-trade costs. - **Detect:** Cost = 0; commission set but slippage = 0; costs not scaled by turnover. - **Why it inflates:** A strategy with a 5 bps gross edge and 200% monthly turnover is *guaranteed* to lose after realistic costs, yet looks profitable frictionless. - **Fix:** Apply realistic **commission + spread + slippage** per fill, scaled by turnover. A useful sanity check: compute the **break-even cost** (the per-trade cost at which the strategy's edge disappears) and ask whether real-world costs exceed it. ```python COST_BPS = 5 # round-trip commission + half-spread + slippage, in basis points df['turnover'] = df['position'].diff().abs() df['net_ret'] = df['gross_ret'] - df['turnover'] * (COST_BPS / 1e4) ``` ### 3.7 Market impact & liquidity ignored *(High)* - **What:** Assumes you can trade any size at the quoted price with no impact. - **Detect:** Position sizes large vs. average daily volume (ADV); thin instruments; no cap on participation rate. - **Why it inflates:** Real orders move the price against you; the backtest captures alpha that evaporates at size. - **Fix:** Cap order size to a fraction of ADV (e.g. ≤1–5% participation), add a size-dependent **impact cost** model (e.g. square-root impact), and stress-test at the capital you actually intend to deploy. ### 3.8 Regime dependence *(High)* - **What:** The strategy only works in one market regime (one long bull market, one low-vol period, one rate environment) present in the backtest. - **Detect:** Backtest spans a single regime; equity curve is one smooth uptrend; no drawdowns through 2008/2020/2022-type stress. - **Why it inflates:** It is a bet on the regime continuing, not a durable edge. - **Fix:** Test across **multiple regimes** (bull/bear/high-vol/low-vol/rate cycles), tag performance by regime, and run **out-of-sample on a different regime** than the one tuned on. ### 3.9 No out-of-sample / walk-forward validation *(Critical)* - **What:** All reported performance is in-sample. - **Detect:** No train/test split; no holdout; no walk-forward. - **Why it inflates:** In-sample performance is an upper bound, not an estimate of the future. - **Fix:** **Walk-forward analysis** — repeatedly tune on a rolling/anchored window and evaluate on the immediately following untouched window; concatenate the out-of-sample segments to get an honest equity curve. Keep a **final lockbox** holdout never used for tuning. ### 3.10 Position sizing & risk-of-ruin *(Critical)* - **What:** Sizing is ad hoc, fixed-fractional too large, or unbounded; no account-level risk limit. - **Detect:** "All-in" sizing; no max position; bet size that risks a large fraction of capital per trade; ignoring the probability of a terminal drawdown. - **Why it inflates / endangers:** Even a positive-edge strategy bankrupts with too-large bets due to the variance drag and path dependence; risk-of-ruin can be high even when expectancy is positive. - **Fix:** **Volatility targeting** (size to a target portfolio vol) and/or a **fractional Kelly** stake (e.g. ¼–½ Kelly, never full Kelly), with hard per-trade and per-account risk caps. Estimate **risk-of-ruin** explicitly given edge, variance, and bet size. ### 3.11 Leverage & path dependence *(High–Critical)* - **What:** Leverage applied to the *return series* without modeling margin, intraday path, or forced liquidation. - **Detect:** Returns simply multiplied by a leverage factor; no margin call / liquidation logic; ignoring volatility drag of leveraged compounding. - **Why it inflates:** A 3× return series ignores that an intraday spike can liquidate you before the favorable close; leveraged compounding suffers volatility decay. - **Fix:** Model **margin requirements, intraday path, and liquidation**; account for **volatility drag**; cap leverage; backtest on the **actual path**, not just close-to-close. ### 3.12 Funding & borrow costs ignored *(High for shorts/crypto/leverage)* - **What:** Short rebate/borrow fees, hard-to-borrow constraints, perpetual-swap **funding rates**, and carry are omitted. - **Detect:** Short strategy with no borrow cost; crypto perp strategy with no funding; long-short with no financing. - **Why it inflates:** Borrow/funding can exceed the gross edge, especially on shorts and crypto perps where funding flips sign and is paid continuously. - **Fix:** Apply **borrow fees** (and hard-to-borrow availability) on shorts and **funding rates** on perps; include financing on leveraged/long-short books. ### 3.13 Timezone / session-boundary bugs *(Medium–High)* - **What:** Bars aligned to the wrong timezone, daylight-saving shifts, or session open/close handled incorrectly; mixing exchange time and UTC. - **Detect:** Naive timestamps; DST not handled; daily bars rolled at the wrong hour; overnight gaps treated as intraday moves; 24/7 crypto vs. session-based equities mismatch. - **Why it inflates:** Misaligned bars can accidentally introduce look-ahead or attribute the overnight gap to an intraday signal. - **Fix:** Use **timezone-aware** timestamps, define sessions explicitly per instrument, handle DST, and verify bar boundaries against the exchange calendar. ### 3.14 Repainting indicators (Pine / charting) *(Critical on the affected signal)* - **What:** An indicator changes its historical values after the fact — `security()` pulling higher-timeframe data without offset/lookahead guards, signals confirmed only on the **closing** bar but evaluated intrabar in the backtest, `request.security(..., lookahead=barmerge.lookahead_on)`. - **Detect:** Pine `request.security` with `lookahead_on`; signals that reference the current (unclosed) bar; alerts that fire then disappear; backtest results that change on reload. - **Why it inflates:** Repainting bakes future information into historical signals — the on-chart backtest is fiction. - **Fix:** Use `lookahead=barmerge.lookahead_off` and reference **confirmed/closed** values (`[1]` offset for HTF data); only act on **bar-close confirmed** signals (`barstate.isconfirmed`); compare strategy-tester results to bar-close-only alerts. ### 3.15 Curve-fit stop-loss / take-profit levels *(Medium–High)* - **What:** Stop and target levels (or trailing parameters) tuned to maximize the same backtest. - **Detect:** Oddly specific levels (e.g. "stop at 2.37×ATR, target 4.81×ATR"); stops/targets swept in the same optimization as entries. - **Why it inflates:** Exits become additional fitted parameters that snugly avoid the specific historical losers. - **Fix:** Derive exits from **risk logic** (volatility/ATR-based, fixed-R multiples) chosen *a priori*; validate stop/target robustness **out-of-sample**; prefer few, round, rule-based levels over swept ones. ### 3.16 Too few trades / weak statistical significance *(Medium–High)* - **What:** The track record rests on too few trades to distinguish skill from luck. - **Detect:** < ~30 trades (often < 100 for a real claim); a handful of trades dominate P&L; concentrated in one period. - **Why it inflates:** Small samples have huge sampling error; one or two lucky trades can manufacture a great Sharpe. - **Fix:** Require a **meaningful trade count**, test across periods, and report **confidence intervals** on the metrics; treat the Sharpe as an estimate with error bars, not a point. --- ## Step 4: Red-Flags Quick Scan Instant danger signs. Any hit warrants a finding and a closer look at the related checklist item. - **Sharpe > 3 on daily data** (or > ~2 for a simple retail strategy) — almost always look-ahead, cost omission, or overfitting. - **Equity curve is "too smooth"** / near-straight diagonal with tiny drawdowns — classic look-ahead or in-sample fitting signature. - **No transaction costs / slippage** anywhere in the code. - **Indicator or signal references the future** — `close[future]`, `shift(-n)`, `lookahead_on`, full-series `fit()` before split. - **Parameters tuned on the whole series** with no holdout. - **< 30 trades**, or P&L dominated by 1–2 trades. - **Win rate ~100%** or suspiciously high with no losing streak. - **Many configs tried, only the best reported** (undisclosed trial count). - **Universe is "today's" index / current tickers** over a long history (survivorship). - **Leverage applied by multiplying returns**, no margin/liquidation logic. - **Shorts/perps with no borrow/funding cost.** - **Backtest results change on reload** (repainting). - **Reported max drawdown is implausibly small** relative to the strategy's volatility. --- ## Step 5: Output Format Produce the review in this structure. ### 5.1 Findings summary table | ID | Issue | Severity | Where | |----|-------|----------|-------| | F1 | Look-ahead: signal uses same-bar close | Critical | `strategy.py:42` | | F2 | No transaction costs modeled | Critical | `backtest.py` (no cost term) | | F3 | Params tuned on full series, no OOS | High | optimizer block | | … | … | … | … | ### 5.2 Per-finding detail For each finding: - **ID & title** (e.g. *F1 — Look-ahead bias in entry signal*). - **Severity** and one-line rationale. - **Impact:** what it does to the reported numbers (e.g. "inflates Sharpe; equity curve is not achievable"). - **How to verify:** the exact check the user can run to confirm (e.g. "lag the signal by one bar and re-run — if returns collapse, this was the source"). - **Fix:** concrete remedy, with a short **before/after** code snippet where useful. ### 5.3 Metrics sanity check Recompute or critically assess the headline metrics; define each so the user can audit them: - **Sharpe ratio** = (annualized mean excess return) ÷ (annualized return volatility). Daily-to-annual scaling uses √252 (√365 or √(24·365) for crypto). Flag any Sharpe that is implausible for the asset/timeframe. - **Sortino ratio** = like Sharpe but the denominator is **downside** deviation only (volatility of negative returns), rewarding strategies that are only "volatile" to the upside. - **Max drawdown** = the largest peak-to-trough decline in the equity curve; check it is measured on the **net** (post-cost) curve and is plausible vs. the strategy's volatility. - **Deflated Sharpe Ratio (DSR)** = the probability the true Sharpe > 0 after correcting the observed Sharpe for the **number of trials**, the **dispersion** of trial Sharpes, and **skew/kurtosis** of returns. Ask for the trial count so DSR can be assessed. - **Probability of Backtest Overfitting (PBO)** = via CSCV, the fraction of splits where the in-sample-best configuration ranks below median out-of-sample; **> 0.5 ⇒ overfit.** - **Trade count & exposure:** number of trades (significance), average holding period, **time in market** (exposure) — low exposure with a high Sharpe can still mean tiny, fragile edges. - **Turnover & break-even cost:** confirm net-of-cost results and that real costs are below the break-even cost. Where you cannot recompute (no return series provided), say so explicitly and state what the user must supply. ### 5.4 Verdict A short, honest bottom line: *which* findings (if any) are **Critical** and therefore block deployment, what must be re-run (e.g. "add costs + lag signals + walk-forward, then re-evaluate"), and whether the remaining edge is plausible. ### 5.5 Disclaimer (always include) > **Educational review — not financial advice.** This is a methodological critique of a > strategy/backtest, not investment advice and not a recommendation to trade. Even after > every fix, **past backtest performance does not guarantee live results.** Markets change, > costs and liquidity vary, and all trading carries risk of loss. You are solely responsible > for your own trading and risk decisions. --- ## Tools that operationalize these fixes These Viprasol open-source tools implement parts of this methodology and pair naturally with this review: - **[`bar-by-bar`](https://github.com/Viprasol-Tech/bar-by-bar)** — an agentic backtester with a built-in **look-ahead guard** that structurally prevents the Step 3.1 / 3.9 class of bugs (point-in-time iteration, no future leakage). - **[`edgehunt`](https://github.com/Viprasol-Tech/edgehunt)** — a prediction-market edge/arbitrage engine for finding and pricing real edges, useful when validating whether a measured edge survives realistic costs (Step 3.6 / 3.7). Use this skill to *find* the failure modes; use those tools to *enforce* the fixes.