Backtest Change

Backtest a change before you ship it

Validate a data-driven change against real historical data before opening the PR — and be willing to abandon or redirect the approach when the data says it won't work. This is the gate that stops a plausible-but-ineffective change from shipping.

Use it for any change whose success is measurable: monitor thresholds, alert routing / re-notify cadence, metric/log/trace queries, sampling rates, cache TTLs, rate limits, autoscaling params, or a perf optimization with a latency/throughput target.

When this fires without being asked

The most valuable case is the one nobody invokes deliberately: someone hands you a number. "Add a monitor at 700 MB warn / 1.2 GB critical." "Set the timeout to 5s." A named threshold arrives with an implicit claim attached — that it separates the bad cases from the good ones. That claim is testable, and it is often wrong, because the person proposing it has seen the incident population and not the healthy population.

Treat a user-supplied number as a hypothesis to test, never as a specification to implement. Then say plainly what the data did to it.

Worked example. A request arrived for a per-host memory monitor at "700 MB warn / 1.2 GB critical", motivated by two hosts that had OOM-wedged at a 2 GiB limit. A 30-day replay found a third host that had held 1.34–1.37 GB flat for ~21 consecutive hours with no incident — so the proposed critical would have paged continuously for most of a day. Shipped 1.6 GB instead. The warning was kept exactly as proposed, because a non-paging tier is allowed to sit inside normal range when its job is lead time. Both halves of that outcome came from the backtest, not from the proposal.

Inputs

You need: the change intended (and the file(s) if known), and the goal it should achieve — which metric/signal should move, in which direction, by how much. If the goal isn't stated, ask. A backtest is meaningless without a target.

When invoked as /backtest-change, parse $ARGUMENTS for the same two things.

The discipline (why this exists)

A change that looks right is not the same as a change the data supports. The common failure is shipping a fix whose premise is wrong — the real driver was something else, so the metric never moves. Catch that before the PR, not in a post-merge validation.

Workflow

1. State the hypothesis precisely. "Changing X will move metric M from ~A to ~B because C." Write it down. If you can't name the metric and the expected direction, stop and clarify.

2. Find the authoritative data source and respect sampling:

   • Metrics (standard Datadog metrics, trace.*, CloudWatch) are ~100% — use these to count rates/volumes/percentiles.
   • Spans and logs are often heavily sampled on the success path; don't count volume from them.
   • For alert/page/incident questions, pull the alert system's own event history (e.g. incident.io alerts), not a proxy.
   • Beware aggregation defaults that hide the shape you're testing — e.g. a scalar query that silently averages a max: series returns avg-of-max and will understate peaks. Set the aggregator explicitly.

3. Pull a representative window (typically 7–30 days; long enough to include the conditions the change targets).

4. Replay old logic vs new logic over that same window. Compute concrete deltas: old N vs new M — alerts fired, pages, error rate, p95, cost, rows, whatever the goal metric is. For threshold/monitor changes, evaluate both the old and the new condition against the historical series and count transitions. Identify which groups/series change, not just the aggregate. For a brand-new monitor, "old" is 0 — say so explicitly rather than omitting it. For a composite (a gate combining, say, a rate condition and a traffic floor), replay the conditions jointly over the same window. Each one alone will fire on windows the composite would have suppressed, so per-condition counts overstate what the gate actually does.

5. Separate the two populations. The threshold's whole job is to divide incident from healthy. Report the highest legitimate value observed and the lowest incident value. If they overlap, the threshold cannot work at any setting and the signal itself needs to change — say that instead of picking a number in the overlap.

6. Classify the result:

   • EFFECTIVE — data shows the change achieves the goal. Capture the old-vs-new numbers for the PR body.
   • PARTIAL — moves the metric but not enough / not for the cases that matter. Note the gap.
   • INEFFECTIVE / PREMISE DISPROVED — the data shows the real driver is elsewhere, or the change barely moves M. Stop. Do not open the PR. Report what the data actually shows and propose the lever that would work.
   • REVISED — the goal is sound but the proposed number isn't. Ship the corrected value and state prominently what you changed and why.

7. Only if it holds up, proceed to the change + PR, and put the backtest in the PR body: the hypothesis, the window, old-vs-new numbers, and a link to the live dashboard/query (prefer a link over stale typed numbers).

Output

A short backtest report:

• Hypothesis and goal metric.
• Window + data source (and any sampling caveat applied).
• Old vs new with hard numbers and which groups changed.
• Population separation — highest healthy value vs lowest incident value.
• Verdict (EFFECTIVE / PARTIAL / INEFFECTIVE / REVISED) + recommendation. If INEFFECTIVE, the alternative lever.

Principles

• Backtest before acting; never claim a change works without replaying data.
• A user-supplied number is a hypothesis. Testing it is the job, not overriding the request — but when the data rejects it, say so and ship the corrected value.
• Be willing to reverse — a disproved premise is a successful backtest, not a failure.
• Prefer dashboard/query links over typed numbers that go stale.
• Distinguish tiers by consequence: a chatty non-paging warning can be acceptable; a chatty page destroys trust in the monitor.
• When the change spans owners (e.g. an external-config change + a repo change), say which half the data supports and which is out of scope.