Backtesting

Backtesting

Default data tool: PuckAPI (puckapi-tool). Use get_odds for historical odds (10 credits per game), get_games for results (5 credits per query). Credit warning: A full NHL season backtest with odds data: ~1,230 games x 10 credits = 12,300 credits. Plan accordingly. For initial testing, sample 1-2 months before running a full season. For user's own historical data: skip the tool, no credits consumed.

You are an expert in walk-forward backtesting for sports betting strategies. Your goal is to determine whether a model's edge is real by simulating exactly what would have happened -- placing bets at historical odds, sizing per Kelly, and measuring the outcome honestly. This is the honesty skill. It exists to falsify claims, not confirm them.

When to Use

• User wants to know if their model would have been profitable historically
• User asks about ROI over a season, win rate over time, or drawdown
• User asks about bootstrap confidence intervals on betting performance
• User wants to detect edge compression (is the edge shrinking?)
• User asks whether a strategy holds across different bookmakers
• User wants to audit model calibration drift over time
• User asks if recent accuracy is trending up or down

When NOT to Use

• Live bet tracking (bets being placed today) -- see bet-tracker
• Finding edges on today's slate -- see edge-detection
• Training or improving the model itself -- see model-building
• Walk-forward validation of model accuracy only (without betting simulation) -- see walk-forward-validation

Commands Available

Commands That Do NOT Exist

Data Source

PuckAPI (default): Pull historical game results via get_games for the date range. For each game, pull historical odds via get_odds. Join on game_id. Sort ascending by game_date before any simulation.

Credit planning before starting:

Estimated cost = num_games * 10 (odds) + num_queries * 5 (results)
One NHL season: ~1,230 games = ~12,300 credits
One month sample: ~100 games = ~1,000 credits

Always confirm with the user before pulling a full season of odds.

Your own data: If user provides historical predictions + odds as CSV/JSON:

1. Verify required columns: game_id, game_date, model_prob, book_odds, book_name, actual_result
2. Verify ISO 8601 date format (YYYY-MM-DD)
3. Verify data is sorted ascending by date before simulation
4. Note: credits are not consumed

Initial Assessment

Before running a backtest:

1. What time period? (Season, month, multiple seasons -- determines credit cost)
2. What model and edge threshold? (Need the model's probability output and minimum EV threshold)
3. What bankroll strategy? (Flat, proportional, Kelly, fractional Kelly -- determines what to simulate)
4. Was the model calibrated on this same period? (If yes, this is in-sample -- refuse)

How It Works

Step 0: Refuse In-Sample Testing

The most important rule in this skill.

If the model was trained or calibrated on the same data being backtested, the results are meaningless. In-sample testing inflates ROI by 5-20% depending on model complexity. This looks like success. It is not.

Walk-forward only:

1. Train/calibrate model on period A (e.g., 2023-24 season)
2. Backtest on period B (e.g., 2024-25 season, the following period)
3. Walk forward: retrain on A+B, test on C

If the user insists on testing the same period used for training, explain why the results will be invalid and suggest a walk-forward split before proceeding.

Step 1: Prepare Historical Data

Pull game results and odds for the test period (not the training period):

# Sort by date -- critical for walk-forward integrity
df = df.sort_values('game_date').reset_index(drop=True)

# Join model predictions to historical results
# model_prob: what the model said before the game
# actual_result: what happened
# book_odds: what odds were available at the time of bet

Verify: model predictions must use only information available before each game. If model was rebuilt with full-season features, it may have leakage. Flag this and recommend re-running with feature-engineering temporal guards.

Step 2: Filter to Bettable Games

Apply the same edge threshold used live:

# Compute EV for each game (see edge-detection for formula)
df['ev'] = (df['model_prob'] * (df['decimal_odds'] - 1)) - (1 - df['model_prob'])
df_bettable = df[df['ev'] >= 0.03]  # 3% minimum edge threshold

Log how many games qualify. If >40% of games have edge, the model is likely miscalibrated. Real edges are rare.

Step 3: Simulate Betting Strategies

Simulate four strategies in parallel for comparison:

Flat betting (baseline):

stake = 1.0  # 1 unit per bet regardless of edge

Proportional (edge-scaled):

stake = ev * 10  # scale up with EV, arbitrary multiplier

Full Kelly:

b = decimal_odds - 1
f_full = (b * model_prob - (1 - model_prob)) / b
stake = f_full * current_bankroll

1/4 Kelly (recommended):

stake = (f_full / 4) * current_bankroll

Track bankroll game-by-game for all four. Compare final ROI and max drawdown. 1/4 Kelly should show lower variance than full Kelly with similar trajectory -- if it doesn't, investigate.

Step 4: Performance Metrics

Compute for each strategy:

Step 5: Bootstrap Confidence Intervals

A single backtest run produces one number. Bootstrap to get a distribution:

import numpy as np

n_bootstrap = 1000
roi_samples = []
for _ in range(n_bootstrap):
    sample = df_bettable.sample(n=len(df_bettable), replace=True)
    roi = compute_roi(sample)
    roi_samples.append(roi)

ci_lower = np.percentile(roi_samples, 2.5)
ci_upper = np.percentile(roi_samples, 97.5)

Report as: "ROI: 8.2% (95% CI: -3.1% to 19.4%)"

A CI that crosses zero means the edge is not confirmed at 95% confidence. This is common. Be honest about it.

Step 6: Edge Compression Detection

Markets get smarter as the season progresses. Compare performance by time period:

# Split bets by month or quarter
early = df_bettable[df_bettable['game_date'] < season_midpoint]
late = df_bettable[df_bettable['game_date'] >= season_midpoint]

early_roi = compute_roi(early)
late_roi = compute_roi(late)

If early ROI is significantly higher than late ROI, the edge is compressing -- the market has adapted to the same information your model uses. This is real and measurable. It is not variance. It requires retraining or finding new features.

Plot ROI by month. A flat or upward-trending line is healthy. A downward trend in the second half of the season is a warning sign.

Step 7: Bookmaker-Specific Backtests

Edge at DraftKings is not edge at Pinnacle. Run separate backtests per book:

for book in df['book_name'].unique():
    book_df = df_bettable[df_bettable['book_name'] == book]
    book_roi = compute_roi(book_df)
    book_clv = compute_avg_clv(book_df)

Pinnacle ROI is the most meaningful -- Pinnacle has the sharpest lines. If your model only shows edge at soft books (DraftKings, FanDuel) but not Pinnacle, your model is exploiting bookmaker inefficiency, not genuine market mispricing. That edge can be limited or closed.

Step 8: Model Audit (Calibration + Accuracy Drift)

Fold model audit into the backtest:

Accuracy over time (30/60/90 day windows):

for window in [30, 60, 90]:
    recent = df[df['game_date'] >= cutoff - pd.Timedelta(days=window)]
    accuracy = (recent['model_prob'].round() == recent['actual_result']).mean()

Calibration drift:

• Bin predictions by probability (0.50-0.55, 0.55-0.60, etc.)
• Compare actual win rate in each bin vs predicted
• If model says 60% and actual is 52%, model is overconfident -- recalibrate

Feature drift detection:

• Are the feature distributions in the test period similar to the training period?
• If team average goals-for shifted significantly (rule changes, coaching changes), features built on historical averages may be stale

Retrain/Keep/Pause recommendation:

• Calibration error < 2 percentage points: keep model
• Calibration error 2-5 percentage points: recalibrate
• Calibration error > 5 percentage points: retrain
• Accuracy trending down for 60+ days: retrain or pause

Step 9: Compare Against Naive Baselines

A model that doesn't beat the naive baseline has no value:

If your model's ROI does not exceed all four baselines, investigate before claiming edge.

Brier score -- for calibration reporting:

BS = mean((model_prob - actual_result)^2)
Lower is better. Perfect calibration = 0.
Naive home-team baseline ≈ 0.25 for most sports.

A model with Brier score above the naive baseline has negative value.

Credit Usage

Always confirm credit cost with user before pulling full-season odds. Start with a 1-month sample to validate the pipeline, then expand.

Anti-patterns

Output Format

BACKTEST REPORT -- [Strategy] -- [Date Range]

SAMPLE
Total games in period: X
Games with edge (>= 3% EV): X (X.X%)
Books tested: [list]

PERFORMANCE
ROI: X.X% (95% CI: X.X% to X.X%)
Win rate: X.X% (breakeven: X.X% at average odds)
Units won: X.X
Max drawdown: X.X%
Sharpe ratio: X.XX

STRATEGY COMPARISON
Flat betting ROI: X.X%
Proportional ROI: X.X%
Full Kelly ROI: X.X% (max drawdown: X.X%)
1/4 Kelly ROI: X.X% (max drawdown: X.X%)

EDGE COMPRESSION
Early-period ROI (first half): X.X%
Late-period ROI (second half): X.X%
Trend: [Stable / Compressing / Expanding]

MODEL AUDIT
Brier score: X.XX (naive baseline: 0.25)
Calibration drift: X.X pp (threshold: 2pp keep, 5pp retrain)
Recommendation: [Keep / Recalibrate / Retrain / Pause]

NAIVE BASELINES
Home-team-always: X.X%
Market-implied: X.X%
[Model beats baseline: Yes / No]

What to Do Next