Predict First

predict-first

Say what will happen. Then measure. Then edit.

When to use this

Use it when the question is "will this change make the numbers better?" and a normal unit test cannot answer it. Typical cases:

• a new filter, rule, or parameter in a trading strategy
• a change to a loss, a feature, or a training schedule in an ML model
• a fix to a physics model, an integrator, or a solver setting
• an optimisation that should make something faster or smaller
• a change to a data pipeline that should change counts or distributions

Do not use it for refactors, API changes, or bugs a test can decide. Say so and skip the protocol.

The protocol

Follow the six steps in order. Do not skip ahead. Do not edit repository files before step 6.

1. Hypothesis

One sentence: which cause produces which symptom, and what the change does about it. If you cannot write it in one sentence, you do not have a hypothesis yet.

2. Predictions, written before the first run

Write a numbered list P1 to Pn. Every prediction needs a number and a threshold. "It gets better" is not a prediction. "Sharpe improves by at least 0.3 on the training window" is.

The list must contain:

• A symptom check. One prediction that the baseline reproduces the problem. If you cannot see the problem before the change, you cannot see it go away.
• A counter-prediction. One thing the change will probably not achieve. Say the weak spot out loud before the run.
• A falsifier. One outcome that, if it happens, means the hypothesis is wrong and the change must not be adopted.
• One prediction per independent symptom. If the cause is real, several symptoms disappear together. Test each one separately.

Then write the decision rule: which predictions must hold for the change to be adopted. Write it now, not after you see the numbers.

3. Patch, do not edit

Write a script in a scratch directory, never in the repository. The script imports the real code and swaps out the one function, argument, or constant under test for the duration of the measurement. In Python use unittest.mock.patch.object. In other languages use whatever gives the same effect: an environment variable, a config override, a feature flag, a shadow copy of one file.

Run the baseline and the patched version on the same inputs, same seeds, same window. Run the baseline first.

If the full run is too slow, reduce it, and say in the report that you did.

Use templates/predict_first_template.py as the starting point.

4. Measure

Run the script. Collect the numbers the predictions ask for and nothing else. Do not start interpreting yet.

5. Verdict

Fill the table, one row per prediction:

ID  Prediction                      Measured   Result
P1  baseline drift > 100 %          4.2e13     holds
P2  patched drift < 1 %             -0.0023    holds
P3  phase error still > 0.01 rad    0.09 rad   holds
P4  amplitude within 5 %            +0.0003    holds
VERDICT: confirmed, adopt the change

Rules for this step:

• Predictions are frozen. Do not reword, loosen, or drop one after the run.
• A failed prediction is reported as failed. Explaining it away is not allowed. "P1 fails, the gain was 0.02 not 0.3" is the whole sentence.
• Apply the decision rule from step 2 as written.
• Report the result to the user in full, including failures, before proposing anything.

6. Only now: act

If the decision rule says adopt, propose the real change. Put the verdict table in the commit message or pull request so the evidence travels with the code. If the rule says do not adopt, say so, keep the scratch script as a record, and go back to step 1 with what you learned.

Hard rules

• No edits to repository files before the verdict. None.
• Predictions before measurements, always, in writing, with numbers.
• At least one counter-prediction and one falsifier per experiment.
• Baseline and patch on identical inputs and seeds.
• Frozen predictions. A failed one stays failed.
• Reduced runs are declared as reduced.
• The user sees the full table, not a summary of the good parts.

Report format

Use this structure when reporting back:

predict-first: <one-line hypothesis>

<table as above>

VERDICT: <confirmed, adopt | not confirmed, do not adopt | partial: ...>
Notes: <reduced run? surprises? what next?>

templates/report.md has the same skeleton for commit messages and pull requests.