Trading Strategy Backtester

Trading Strategy Backtester

Overview

Backtesting evaluates a trading strategy on historical data. The core pitfall is overfitting — a strategy that looks brilliant historically but fails live. Every design decision must guard against it.

When to Use

• Implementing a new strategy and evaluating its historical performance
• Converting a hypothesis into a testable system
• Comparing multiple strategies with statistical rigour
• Generating walk-forward validation reports
• Diagnosing why a live strategy underperforms its backtest

When NOT to use: For live execution logic — use equity-broker-execution for equities (or crypto-exchange-integration for crypto) instead.

Core Pitfalls

Walk-Forward Validation

from vectorbt.portfolio.base import Portfolio
import pandas as pd
import numpy as np

def walk_forward_test(price_data, signal_fn, n_splits=5, train_pct=0.7):
    """N-fold walk-forward. Returns OOS metrics only."""
    results = []
    fold_size = len(price_data) // n_splits

    for i in range(n_splits):
        start = i * fold_size
        end = start + fold_size if i < n_splits - 1 else len(price_data)
        fold = price_data.iloc[start:end]
        split = int(len(fold) * train_pct)
        train, test = fold.iloc[:split], fold.iloc[split:]

        # Optimise params on train only — DO NOT look at test
        signals_test = signal_fn(test)
        pf = Portfolio.from_signals(
            test['close'], signals_test['entries'], signals_test['exits'],
            fees=0.001, slippage=0.0005
        )
        results.append({
            'fold': i,
            'sharpe': pf.sharpe_ratio(),
            'max_dd': pf.max_drawdown(),
            'total_return': pf.total_return(),
        })

    return {
        'folds': results,
        'mean_sharpe': np.mean([r['sharpe'] for r in results]),
        'consistency': sum(1 for r in results if r['total_return'] > 0) / n_splits,
    }

Transaction Cost Model

# Realistic crypto costs
FEES = {'binance_taker': 0.001, 'kraken_taker': 0.0026}

def estimate_slippage(order_size_usd, avg_volume_usd):
    """Square-root market impact model."""
    return 0.001 * ((order_size_usd / avg_volume_usd) ** 0.5)

pf = Portfolio.from_signals(
    close=prices, entries=long_signals, exits=exit_signals,
    fees=0.001, slippage=0.0005, size=0.95,
)

Statistical Significance

from scipy import stats
import numpy as np

def significance_test(strategy_returns, benchmark_returns, alpha=0.05):
    t_stat, p_value = stats.ttest_ind(strategy_returns, benchmark_returns)
    n = len(strategy_returns)
    sr = strategy_returns.mean() / strategy_returns.std() * np.sqrt(252)
    sr_se = np.sqrt((1 + 0.5 * sr**2) / n)
    sr_p = 2 * (1 - stats.norm.cdf(abs(sr / sr_se)))
    return {'p_value': p_value, 'sharpe': sr, 'sharpe_p': sr_p,
            'significant': p_value < alpha}

def bonferroni_threshold(n_strategies, alpha=0.05):
    return alpha / n_strategies  # adjust for multiple comparisons

Freqtrade Strategy Pattern

from freqtrade.strategy import IStrategy, DecimalParameter
from pandas import DataFrame

class ExampleStrategy(IStrategy):
    buy_rsi  = DecimalParameter(20, 40, default=30, space='buy')
    sell_rsi = DecimalParameter(60, 80, default=70, space='sell')
    minimal_roi = {"0": 0.02}
    stoploss = -0.05

    def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
        dataframe['rsi'] = ta.RSI(dataframe['close'], timeperiod=14)
        return dataframe

    def populate_entry_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
        dataframe.loc[(dataframe['rsi'] < self.buy_rsi.value) &
                      (dataframe['volume'] > 0), 'enter_long'] = 1
        return dataframe

Quick Reference — Key Metrics

Overfitting Red Flags

• OOS Sharpe < 0.5 × IS Sharpe
• Fewer than 30 trades per year
• Sharpe drops >50% on adjacent time windows
• More than 5 free parameters
• Strategy works on only 1 instrument

Common Mistakes

1. Optimising on full dataset — hold out a final test set never touched during development
2. Ignoring crypto funding rates — major cost on multi-day perpetual holds
3. Using close for entry price — entry is next bar open
4. Testing one ticker only — results rarely generalise
5. Forgetting survivorship bias — use point-in-time universe

Anti-Patterns