Backtest Flow

Backtest Flow

Seat check — read this first

The routines below are shared: run them from any seat. What differs is not the tooling but the judgement:

• You are the chat (Condor) → follow the whole flow below. It is intake only: the depth — window choice, metric thresholds, parameter sweeps, overfitting checks, go/no-go — belongs to the directional_trader agent, whose backtesting playbook covers it. Delegate rather than improvising it.
• You are directional_trader → backtesting is YOUR domain. Use your own backtesting playbook; do not hand the judgement to another agent. Steps 1–2 below are still a good intake checklist for what to pin down before running anything. For a long grid you do not want to run turn-by-turn, spawn a background copy of yourself — delegate(action="start", agent="directional_trader", task="...") — and let it run the sweep unattended. That is still you: same playbooks, same thresholds.
• You are any other agent → you can run these, and for a single backtest that is the right move. For a sweep, an overfitting check or a deploy decision, hand it over with delegate(action="start", agent="directional_trader", task="...").

Step 1 — Show available controllers first

Before asking for any parameters, call manage_controllers(action="list", controller_type="directional_trading") and present what's available (controllers + their saved configs). Ask whether the user wants an existing config or a new one.

Step 2 — Get parameters

Once the user picks a config (or decides to create one), ask for:

• Date range — suggest 3 months as a default
• Resolution — 1m, 5m, 15m, 1h (1m is the finest; 1s only on Binance spot)
• Trade cost — see the cost section below; 0.0006 is the safe default

Step 3 — Run

• Single config → run the backtest_chart routine directly
• Multiple configs, a parameter sweep, or "which is best?" → run it as a background task: delegate(action="start", agent="directional_trader", task="..."). It runs them, applies the stability/overfitting rules, and pings the user when done. From another seat that is a hand-off to the specialist; from directional_trader's own seat it is a background copy of itself — same call either way.

Step 4 — Compare (optional)

Every backtest is saved, whoever ran it — use the backtest_compare routine to overlay PnL curves and rank by metrics.

The tool contract

This section is the single copy. Every backtesting playbook links here rather than restating it.

There is exactly one way to run a backtest: the shared backtest_chart routine. It runs the backtest, saves it, charts it, and hands back the metrics as data. Dates are YYYY-MM-DD strings, not epoch seconds.

# Blocking — one short run you are waiting on interactively.
manage_routines(action="run", name="backtest_chart", config={
    "config_name": ..., "start_date": "2025-04-22", "end_date": "2025-07-22",
    "resolution": "1m", "trade_cost": 0.0006, "chart": False,
})

# Fire and forget — long windows and every sweep. submit → read back.
manage_routines(action="run_async", name="backtest_chart", config={...})  # → instance_id
manage_routines(action="get_instance", name=<instance_id>)

Blocking vs fire-and-forget

Both are the same routine — the difference is only whether you wait.

• action="run" gives up after ~2 minutes and hands you the instance_id; the run itself keeps going. It is not a failure, it is a handle.
• action="run_async" → action="get_instance" for long windows and every sweep. Submit the whole grid, then read the instances back; do not serialize a 15-variant sweep through blocking calls.

get_instance returns the finished run in full — the metrics row in table_data, the summary in text, the error text on failure, and the status otherwise. You never need to re-run a window just to see its numbers.

Read the numbers from table_data

One row per run, never parsed from text. N runs concatenate into one table.

The parameter columns are there so a metric is never quoted without the window, resolution and cost it was measured under — the row is self-describing, so you never have to remember to record them separately.

Set chart=False for every sweep — otherwise each run pushes an image into the user's chat. The chart still reaches the web report; a single run you are presenting is the one case worth leaving chart=True.

Every run is saved

The routine stores each completed backtest under its server-side task_id, whoever ran it and from wherever. That is what makes the rest possible:

• re-render a past run with backtest_chart config={"task_id": ...} — no re-run, no API call;
• rank past runs against each other with backtest_compare (2–6 runs, overlaid curves + a ranked table);
• a run from the chat, the dashboard or an agent is one record, not three.

The task_id is in the metrics row and in the summary text. Record it.

Retention: the Hummingbot API archives finished results to bots/data/backtests/{task_id}.json.gz and keeps a count of results — 100 by default via BACKTESTING_MAX_RESULTS (BACKTESTING_RESULTS_PATH sets the directory). Results survive an API restart. Condor's own copy is independent of that reaping, so a run you saved stays renderable and comparable after the API has dropped it.

Config variants are created with manage_controllers(action="upsert", target="config", config_name=..., config_data={...}), adding confirm_override=True when overwriting.

Rules that hold for every strategy family

These are not directional-specific. They apply to any backtest, run by any agent.

Trade cost — the easiest way to fake an edge

trade_cost is a decimal fraction of notional.

Per-exchange reference (one leg):

Use the taker rate unless the strategy is provably passive. Understating cost is the single easiest way to manufacture a profitable backtest that loses money live — if the edge disappears between 0.0002 and 0.0006, there was no edge.

Resolution is fill fidelity, not the candle interval

backtesting_resolution is the granularity the engine simulates fills at, not the controller's candle interval.

• 1m — maximum fidelity, slowest. Use it for the baseline and the final validation run.
• Match the controller interval (15m, 1h) for speed during a wide sweep, then re-run the winner at 1m before deploying.
• A winner that only survives at coarse resolution is not a winner — the gap is usually intrabar stop/TP ordering.

Trade count is a validity gate, not a statistic

Under ~20 executors, no metric in the report means anything — a Sharpe over 8 trades is noise with a decimal point. The routine says so itself: a thin run is flagged above its own numbers in the summary and carries a Trades KPI marked below the gate. Widen the window or loosen the filters and re-run before reading anything into the metrics.

Sweeping and deploying

• Never sweep every parameter at once — that is curve fitting with extra steps. One parameter at a time, looking for a plateau.
• Stability beats the peak. An isolated Sharpe spike surrounded by collapse is overfit; a plateau is a result.
• Never deploy on in-sample numbers alone — a held-out window is mandatory. Hold out the most recent ~30 days from the sweep and keep it clean.
• Report the numbers you actually got, including the bad ones. A NO-GO is a successful outcome, not a failed one.

What is NOT shared

Metric thresholds — what Sharpe is acceptable, what win rate is a red flag, how long a trade should last — are calibrated per strategy family and do not transfer. A 95% win rate at profit factor 1.1 is a red flag for a directional strategy and the normal shape of a funding-arb trade. Read the thresholds from the agent that owns the family (directional_trader → backtesting/interpret_metrics.md), or state that you don't have calibrated ones. Do not borrow another family's table.