# Strategy Validation > End-to-end strategy validation — backtest tearsheet (Sharpe, Sortino, Calmar, VaR), walk-forward optimization, Monte Carlo stress testing, parameter sensitivity heatmaps, and strategy A/B testing. Use for validate strategy, backtest report, walk-forward, Monte Carlo test, parameter sensitivity, or any strategy validation. - Skill: `mahmoud20138/strategy-validation` (Agent Skill) - Install (CLI): `npx skillmds@latest add mahmoud20138/strategy-validation` - Raw SKILL.md: https://api.skillmd.com/api/skills/mahmoud20138/strategy-validation/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Docs & Writing - Author: mahmoud20138 (https://skillmd.com/u/mahmoud20138) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/mahmoud20138/strategy-validation --- # Strategy Validation — Tearsheet, WFO, Monte Carlo, Sensitivity, A/B Testing ## Overview End-to-end strategy validation pipeline: 1. **Backtest Tearsheet** — Sharpe, Sortino, Calmar, VaR, CVaR, Win Rate, HTML report 2. **Walk-Forward Optimization** — anchored and rolling WFO with OOS validation 3. **Monte Carlo Stress Testing** — bootstrap, parameter perturbation, regime shuffle 4. **Parameter Sensitivity** — 2D grid sweep, robustness heatmaps, one-at-a-time 5. **Strategy A/B Testing** — paired t-test, Wilcoxon, KS test, Jobson-Korkie Sharpe diff --- ## Section 1: Backtest Tearsheet ```python import pandas as pd import numpy as np from scipy import stats from datetime import datetime from typing import Optional def compute_tearsheet(equity_curve: pd.Series, returns: pd.Series, trades_df: Optional[pd.DataFrame] = None, benchmark_returns: Optional[pd.Series] = None, risk_free_rate: float = 0.04) -> dict: """Compute comprehensive strategy tearsheet metrics.""" annual_factor = 252 total_return = (equity_curve.iloc[-1] / equity_curve.iloc[0]) - 1 years = len(returns) / annual_factor cagr = (1 + total_return) ** (1 / max(years, 0.01)) - 1 peak = equity_curve.cummax() dd = (equity_curve - peak) / peak max_dd = dd.min() dd_durations = [] in_dd = False; start = None for i, d in enumerate(dd): if d < 0 and not in_dd: in_dd = True; start = i elif d == 0 and in_dd: in_dd = False; dd_durations.append(i - start) vol = returns.std() * np.sqrt(annual_factor) sharpe = (returns.mean() * annual_factor - risk_free_rate) / max(vol, 1e-10) downside_ret = returns[returns < 0] sortino = ((returns.mean() * annual_factor - risk_free_rate) / (downside_ret.std() * np.sqrt(annual_factor)) if len(downside_ret) > 0 else 0) calmar = cagr / abs(max_dd) if max_dd != 0 else 0 var_95 = returns.quantile(0.05) cvar_95 = returns[returns <= var_95].mean() trade_stats = {} if trades_df is not None and not trades_df.empty: closed = trades_df[trades_df["pnl_pips"].notna()] wins = closed[closed["pnl_pips"] > 0]; losses = closed[closed["pnl_pips"] <= 0] trade_stats = { "total_trades": len(closed), "win_rate": round(len(wins) / max(len(closed), 1) * 100, 1), "avg_win": round(wins["pnl_pips"].mean(), 1) if len(wins) > 0 else 0, "avg_loss": round(losses["pnl_pips"].mean(), 1) if len(losses) > 0 else 0, "profit_factor": round(wins["pnl_usd"].sum() / abs(losses["pnl_usd"].sum()), 2) if len(losses) > 0 and losses["pnl_usd"].sum() != 0 else float("inf"), "expectancy_pips": round(closed["pnl_pips"].mean(), 2), "largest_win": round(wins["pnl_pips"].max(), 1) if len(wins) > 0 else 0, "largest_loss": round(losses["pnl_pips"].min(), 1) if len(losses) > 0 else 0, } return { "summary": {"total_return": round(total_return * 100, 2), "cagr": round(cagr * 100, 2), "sharpe": round(sharpe, 3), "sortino": round(sortino, 3), "calmar": round(calmar, 3), "volatility": round(vol * 100, 2), "max_drawdown": round(max_dd * 100, 2), "avg_drawdown_duration": round(np.mean(dd_durations), 0) if dd_durations else 0, "max_drawdown_duration": max(dd_durations) if dd_durations else 0, "var_95": round(var_95 * 100, 4), "cvar_95": round(cvar_95 * 100, 4)}, "trade_stats": trade_stats, "period": f"{equity_curve.index[0]} → {equity_curve.index[-1]}", "bars": len(returns), "years": round(years, 2), } def generate_html_report(tearsheet: dict, monte_carlo: dict, strategy_name: str = "Strategy") -> str: """Generate a standalone HTML report with Chart.js.""" return f"""{strategy_name} Backtest Report

{strategy_name} — Backtest Report

Generated: {datetime.utcnow().strftime('%Y-%m-%d %H:%M UTC')}

Performance Summary

{tearsheet['summary']['total_return']}%
Total Return
{tearsheet['summary']['cagr']}%
CAGR
{tearsheet['summary']['sharpe']}
Sharpe
{tearsheet['summary']['sortino']}
Sortino
{tearsheet['summary']['max_drawdown']}%
Max DD

Trade Statistics

{''.join(f"" for k, v in tearsheet.get('trade_stats', {}).items())}
MetricValue
{k}{v}
⚠ DISCLAIMER: Past performance does not guarantee future results. Walk-forward out-of-sample validation required before live deployment.
""" ``` --- ## Section 2: Walk-Forward Optimization ```python from typing import Callable class WalkForwardOptimizer: @staticmethod def anchored_wfo(data: pd.DataFrame, strategy_fn: Callable, optimize_fn: Callable, train_pct: float = 0.7, n_folds: int = 5) -> dict: """Anchored WFO: training window grows, test window is fixed.""" total = len(data); test_size = total // (n_folds + 1); results = [] for fold in range(n_folds): train_end = total - test_size * (n_folds - fold) test_end = train_end + test_size train = data.iloc[:train_end]; test = data.iloc[train_end:test_end] best_params = optimize_fn(train) oos_returns = strategy_fn(test, best_params) sharpe = (oos_returns.mean() / oos_returns.std()) * np.sqrt(252) if oos_returns.std() > 0 else 0 results.append({"fold": fold, "train_size": len(train), "test_size": len(test), "params": best_params, "oos_sharpe": round(sharpe, 3), "oos_return": round(oos_returns.sum() * 100, 2), "oos_trades": len(oos_returns[oos_returns != 0])}) avg_oos_sharpe = np.mean([r["oos_sharpe"] for r in results]) stabilities = [] params_list = [r["params"] for r in results if isinstance(r["params"], dict)] if params_list: for key in params_list[0]: vals = [p.get(key) for p in params_list if isinstance(p.get(key), (int, float))] if vals and np.mean(vals) != 0: stabilities.append(max(1 - np.std(vals) / abs(np.mean(vals)), 0)) param_stability = round(np.mean(stabilities), 3) if stabilities else 0 return { "method": "anchored_walk_forward", "n_folds": n_folds, "fold_results": results, "avg_oos_sharpe": round(avg_oos_sharpe, 3), "param_stability": param_stability, "verdict": "ROBUST" if avg_oos_sharpe > 0.5 and param_stability > 0.6 else "MARGINAL" if avg_oos_sharpe > 0 else "FAILED — strategy does not generalize", } @staticmethod def rolling_wfo(data: pd.DataFrame, strategy_fn: Callable, optimize_fn: Callable, train_bars: int = 500, test_bars: int = 100) -> dict: """Rolling WFO: fixed-size training window moves forward.""" results = [] for start in range(0, len(data) - train_bars - test_bars, test_bars): train = data.iloc[start:start + train_bars] test = data.iloc[start + train_bars:start + train_bars + test_bars] best_params = optimize_fn(train) oos_returns = strategy_fn(test, best_params) sharpe = (oos_returns.mean() / oos_returns.std()) * np.sqrt(252) if oos_returns.std() > 0 else 0 results.append({"fold": len(results), "params": best_params, "oos_sharpe": round(sharpe, 3)}) return {"method": "rolling_walk_forward", "n_folds": len(results), "avg_oos_sharpe": round(np.mean([r["oos_sharpe"] for r in results]), 3), "fold_results": results} ``` --- ## Section 3: Monte Carlo Stress Testing ```python class MonteCarloStressTester: @staticmethod def monte_carlo_simulation(returns: pd.Series, n_simulations: int = 1000, n_periods: int = 252, initial_capital: float = 10000) -> dict: np.random.seed(42) all_paths = np.zeros((n_simulations, n_periods)) for sim in range(n_simulations): sampled = np.random.choice(returns.values, size=n_periods, replace=True) all_paths[sim] = initial_capital * np.cumprod(1 + sampled) final_values = all_paths[:, -1] max_drawdowns = [] for path in all_paths: peak = np.maximum.accumulate(path) max_drawdowns.append((path - peak).min() / peak.max()) return { "n_simulations": n_simulations, "median_final": round(np.median(final_values), 2), "p5_final": round(np.percentile(final_values, 5), 2), "p95_final": round(np.percentile(final_values, 95), 2), "prob_profit": round((final_values > initial_capital).mean() * 100, 1), "prob_ruin_50pct": round((final_values < initial_capital * 0.5).mean() * 100, 1), "worst_case_dd": round(min(max_drawdowns) * 100, 2), "median_max_dd": round(np.median(max_drawdowns) * 100, 2), } @staticmethod def parameter_perturbation(strategy_fn, base_params: dict, data: pd.DataFrame, perturbation_pct: float = 0.1, n_tests: int = 100) -> dict: results = [] for _ in range(n_tests): perturbed = {} for k, v in base_params.items(): if isinstance(v, (int, float)): delta = v * perturbation_pct * np.random.uniform(-1, 1) perturbed[k] = type(v)(v + delta) else: perturbed[k] = v try: ret = strategy_fn(data, perturbed) sharpe = (ret.mean() / ret.std()) * np.sqrt(252) if ret.std() > 0 else 0 results.append({"params": perturbed, "sharpe": sharpe}) except: results.append({"params": perturbed, "sharpe": -999}) sharpes = [r["sharpe"] for r in results if r["sharpe"] > -999] return { "type": "parameter_perturbation", "n_tests": n_tests, "mean_sharpe": round(np.mean(sharpes), 3), "std_sharpe": round(np.std(sharpes), 3), "pct_profitable": round(sum(1 for s in sharpes if s > 0) / max(len(sharpes), 1) * 100, 1), "verdict": "ROBUST" if np.mean(sharpes) > 0.3 else "FRAGILE — parameter-sensitive", } ``` --- ## Section 4: Parameter Sensitivity ```python import matplotlib; matplotlib.use("Agg") import matplotlib.pyplot as plt import io, base64 class ParameterSensitivity: @staticmethod def grid_sweep_2d(strategy_fn, data: pd.DataFrame, param_a: dict, param_b: dict, fixed_params: dict = None) -> dict: """Sweep two parameters, compute Sharpe at each combination.""" fixed = fixed_params or {} results = np.zeros((len(param_a["values"]), len(param_b["values"]))) for i, va in enumerate(param_a["values"]): for j, vb in enumerate(param_b["values"]): params = {**fixed, param_a["name"]: va, param_b["name"]: vb} try: ret = strategy_fn(data, params) results[i, j] = (ret.mean() / ret.std()) * np.sqrt(252) if ret.std() > 0 else 0 except: results[i, j] = np.nan best_idx = np.unravel_index(np.nanargmax(results), results.shape) peak = results[best_idx] neighbors = results[max(0, best_idx[0]-1):best_idx[0]+2, max(0, best_idx[1]-1):best_idx[1]+2] flatness = 1 - np.nanstd(neighbors) / max(abs(peak), 0.01) return { "param_a": param_a["name"], "param_b": param_b["name"], "matrix": results.tolist(), "best": {param_a["name"]: param_a["values"][best_idx[0]], param_b["name"]: param_b["values"][best_idx[1]], "sharpe": round(peak, 3)}, "flatness": round(flatness, 3), "robust": flatness > 0.7, "note": "Flat surface = robust. Spiky surface = overfit." if flatness < 0.5 else "Good parameter stability.", } @staticmethod def render_heatmap(result: dict, save_path: str = None) -> str: fig, ax = plt.subplots(figsize=(10, 8), facecolor="#131722") ax.set_facecolor("#131722") matrix = np.array(result["matrix"]) im = ax.imshow(matrix, cmap="RdYlGn", aspect="auto", interpolation="nearest") ax.set_xlabel(result["param_b"], color="#e0e0e0") ax.set_ylabel(result["param_a"], color="#e0e0e0") ax.set_title(f"Sharpe Sensitivity: {result['param_a']} vs {result['param_b']}", color="#e0e0e0") plt.colorbar(im, label="Sharpe Ratio") ax.tick_params(colors="#787b86") buf = io.BytesIO() fig.savefig(buf, format="png", dpi=150, bbox_inches="tight", facecolor="#131722") plt.close(fig); buf.seek(0) if save_path: with open(save_path, "wb") as f: f.write(buf.read()) buf.seek(0) return base64.b64encode(buf.read()).decode("utf-8") ``` --- ## Section 5: Strategy A/B Testing ```python class StrategyABTester: @staticmethod def compare(returns_a: pd.Series, returns_b: pd.Series, name_a: str = "Strategy A", name_b: str = "Strategy B", risk_free: float = 0.04/252) -> dict: def metrics(r, name): sharpe = (r.mean() - risk_free) / r.std() * np.sqrt(252) if r.std() > 0 else 0 equity = (1 + r).cumprod() max_dd = ((equity / equity.cummax()) - 1).min() return {"name": name, "total_return": round((equity.iloc[-1] - 1) * 100, 2), "sharpe": round(sharpe, 3), "max_drawdown": round(max_dd * 100, 2), "win_rate": round((r > 0).mean() * 100, 1)} m_a = metrics(returns_a, name_a); m_b = metrics(returns_b, name_b) n = min(len(returns_a), len(returns_b)) t_stat, t_pval = stats.ttest_rel(returns_a.values[:n], returns_b.values[:n]) try: w_stat, w_pval = stats.wilcoxon(returns_a.values[:n], returns_b.values[:n]) except: w_stat, w_pval = 0, 1.0 ks_stat, ks_pval = stats.ks_2samp(returns_a.values, returns_b.values) better = name_a if m_a["sharpe"] > m_b["sharpe"] else name_b return { "strategy_a": m_a, "strategy_b": m_b, "winner": better, "statistically_significant": t_pval < 0.05, "tests": {"paired_ttest": {"t_stat": round(t_stat, 3), "p_value": round(t_pval, 4)}, "wilcoxon": {"p_value": round(float(w_pval), 4)}, "ks_test": {"p_value": round(ks_pval, 4)}}, "verdict": f"{better} is {'SIGNIFICANTLY' if t_pval < 0.05 else 'NOT significantly'} better (p={t_pval:.4f})", } ``` ## Validation Workflow ``` 1. compute_tearsheet() ← Sharpe, Sortino, Calmar, VaR, CVaR 2. monte_carlo_simulation() ← Probability of profit, ruin risk 3. WalkForwardOptimizer ← OOS: does the edge generalize? 4. parameter_perturbation() ← Robustness: ±10% parameters = still works? 5. grid_sweep_2d() + heatmap ← Flat surface = robust, spiky = overfit 6. StrategyABTester.compare() ← Is variant A statistically better? 7. generate_html_report() ← Publish with disclaimer ``` | Test | Pass Criteria | |------|--------------| | OOS Sharpe (WFO) | > 0.5 across folds | | Parameter Stability | > 0.6 across folds | | Bootstrap Prob Profit | > 80% | | Perturbation Robustness | > 70% profitable | | Heatmap Flatness | > 0.7 | | A/B p-value | < 0.05 | --- ## Sharpe Ratio as Validation Metric (Wall Street Quants) > Source: "The Sharpe Ratio Explained (by a quant trader)" by Wall Street Quants (Aug 2024) - **t-statistic = SR * sqrt(N trading days)** — the Sharpe ratio directly determines the statistical significance of a strategy's returns. A higher SR over more trading days yields a larger t-stat, making it easier to reject the null hypothesis that returns equal zero. - A strategy's SR automatically **rank-orders it against alternatives** by the confidence that returns are greater than zero. No separate hypothesis test is needed; SR itself encodes that ranking. - **Benchmarks**: S&P 500 ~0.45, Buffett ~0.75, good hedge funds 2.0+, near-arbitrage strategies ~20. - **Low-SR high-return strategies are NOT superior** to high-SR low-return strategies, because leverage preserves the Sharpe ratio while scaling returns. A strategy with SR 2.0 and 5% return can be levered to 20% return while maintaining SR 2.0, whereas a strategy with SR 0.3 and 20% return cannot improve its SR through leverage. - **When comparing strategies**: prefer the higher SR, then lever to the desired return level. This principle makes SR the single most important metric for strategy selection in quantitative finance.