modbender Backtest Expert
Systematic approach to backtesting trading strategies based on professional methodology that prioritizes robustness over optimistic results.
Core Philosophy
Goal: Find strategies that "break the least", not strategies that "profit the most" on paper.
Principle: Add friction, stress test assumptions, and see what survives. If a strategy holds up under pessimistic conditions, it's more likely to work in live trading.
When to Use This Skill
Use this skill when:
- Developing or validating systematic trading strategies
- Evaluating whether a trading idea is robust enough for live implementation
- Troubleshooting why a backtest might be misleading
- Learning proper backtesting methodology
- Avoiding common pitfalls (curve-fitting, look-ahead bias, survivorship bias)
- Assessing parameter sensitivity and regime dependence
- Setting realistic expectations for slippage and execution costs
Backtesting Workflow
1. State the Hypothesis
Define the edge in one sentence.
Example: "Stocks that gap up >3% on earnings and pull back to previous day's close within first hour provide mean-reversion opportunity."
If you can't articulate the edge clearly, don't proceed to testing.
2. Codify Rules with Zero Discretion
Define with complete specificity:
- Entry: Exact conditions, timing, price type
- Exit: Stop loss, profit target, time-based exit
- Position sizing: Fixed $$, % of portfolio, volatility-adjusted
- Filters: Market cap, volume, sector, volatility conditions
- Universe: What instruments are eligible
Critical: No subjective judgment allowed. Every decision must be rule-based and unambiguous.
3. Run Initial Backtest
Test over:
- Minimum 5 years (preferably 10+)
- Multiple market regimes (bull, bear, high/low volatility)
- Realistic costs: Commissions + conservative slippage
Examine initial results for basic viability. If fundamentally broken, iterate on hypothesis.
4. Stress Test the Strategy
This is where 80% of testing time should be spent.
Parameter sensitivity:
- Test stop loss at 50%, 75%, 100%, 125%, 150% of baseline
- Test profit target at 80%, 90%, 100%, 110%, 120% of baseline
- Vary entry/exit timing by ±15-30 minutes
- Look for "plateaus" of stable performance, not narrow spikes
Execution friction:
- Increase slippage to 1.5-2x typical estimates
- Model worst-case fills (buy at ask+1 tick, sell at bid-1 tick)
- Add realistic order rejection scenarios
- Test with pessimistic commission structures
Time robustness:
- Analyze year-by-year performance
- Require positive expectancy in majority of years
- Ensure strategy doesn't rely on 1-2 exceptional periods
- Test in different market regimes separately
Sample size:
- Absolute minimum: 30 trades
- Preferred: 100+ trades
- High confidence: 200+ trades
5. Out-of-Sample Validation
Walk-forward analysis:
- Optimize on training period (e.g., Year 1-3)
- Test on validation period (Year 4)
- Roll forward and repeat
- Compare in-sample vs out-of-sample performance
Warning signs:
- Out-of-sample <50% of in-sample performance
- Need frequent parameter re-optimization
- Parameters change dramatically between periods
6. Evaluate Results
Questions to answer:
- Does edge survive pessimistic assumptions?
- Is performance stable across parameter variations?
- Does strategy work in multiple market regimes?
- Is sample size sufficient for statistical confidence?
- Are results realistic, not "too good to be true"?
Decision criteria:
- ✅ Deploy: Survives all stress tests with acceptable performance
- 🔄 Refine: Core logic sound but needs parameter adjustment
- ❌ Abandon: Fails stress tests or relies on fragile assumptions
Key Testing Principles
Punish the Strategy
Add friction everywhere:
- Commissions higher than reality
- Slippage 1.5-2x typical
- Worst-case fills
- Order rejections
- Partial fills
Rationale: Strategies that survive pessimistic assumptions often outperform in live trading.
Seek Plateaus, Not Peaks
Look for parameter ranges where performance is stable, not optimal values that create performance spikes.
Good: Strategy profitable with stop loss anywhere from 1.5% to 3.0% Bad: Strategy only works with stop loss at exactly 2.13%
Stable performance indicates genuine edge; narrow optima suggest curve-fitting.
Test All Cases, Not Cherry-Picked Examples
Wrong approach: Study hand-picked "market leaders" that worked Right approach: Test every stock that met criteria, including those that failed
Selective examples create survivorship bias and overestimate strategy quality.
Separate Idea Generation from Validation
Intuition: Useful for generating hypotheses Validation: Must be purely data-driven
Never let attachment to an idea influence interpretation of test results.
Common Failure Patterns
Recognize these patterns early to save time:
- Parameter sensitivity: Only works with exact parameter values
- Regime-specific: Great in some years, terrible in others
- Slippage sensitivity: Unprofitable when realistic costs added
- Small sample: Too few trades for statistical confidence
- Look-ahead bias: "Too good to be true" results
- Over-optimization: Many parameters, poor out-of-sample results
See references/failed_tests.md for detailed examples and diagnostic framework.
Available Reference Documentation
Methodology Reference
File: references/methodology.md
When to read: For detailed guidance on specific testing techniques.
Contents:
- Stress testing methods
- Parameter sensitivity analysis
- Slippage and friction modeling
- Sample size requirements
- Market regime classification
- Common biases and pitfalls (survivorship, look-ahead, curve-fitting, etc.)
Failed Tests Reference
File: references/failed_tests.md
When to read: When strategy fails tests, or learning from past mistakes.
Contents:
- Why failures are valuable
- Common failure patterns with examples
- Case study documentation framework
- Red flags checklist for evaluating backtests
Critical Reminders
Time allocation: Spend 20% generating ideas, 80% trying to break them.
Context-free requirement: If strategy requires "perfect context" to work, it's not robust enough for systematic trading.
Red flag: If backtest results look too good (>90% win rate, minimal drawdowns, perfect timing), audit carefully for look-ahead bias or data issues.
Tool limitations: Understand your backtesting platform's quirks (interpolation methods, handling of low liquidity, data alignment issues).
Statistical significance: Small edges require large sample sizes to prove. 5% edge per trade needs 100+ trades to distinguish from luck.
Discretionary vs Systematic Differences
This skill focuses on systematic/quantitative backtesting where:
- All rules are codified in advance
- No discretion or "feel" in execution
- Testing happens on all historical examples, not cherry-picked cases
- Context (news, macro) is deliberately stripped out
Discretionary traders study differently—this skill may not apply to setups requiring subjective judgment.