# Predict First > Use before changing code whose success is a measurement, not a passing test - trading strategies and backtests, ML models, simulations, numerical solvers, performance tuning, data pipelines. Makes you write numeric predictions first (including one against the change), try the change as a temporary patch without editing the repo, and judge it against the predictions before any edit. Trigger on "does this fix work", "will this improve", "let's try", "backtest", "tune", "the numbers are off", or any proposed change to a model, strategy, or solver. - Skill: `flowbudget/predict-first` (Agent Skill, multi-file: 3 files) - Install (CLI): `npx skillmds@latest add flowbudget/predict-first` - Raw SKILL.md: https://api.skillmd.com/api/skills/flowbudget/predict-first/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: Flowbudget (https://skillmd.com/u/flowbudget) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/flowbudget/predict-first --- # predict-first Say what will happen. Then measure. Then edit. ## When to use this Use it when the question is "will this change make the numbers better?" and a normal unit test cannot answer it. Typical cases: - a new filter, rule, or parameter in a trading strategy - a change to a loss, a feature, or a training schedule in an ML model - a fix to a physics model, an integrator, or a solver setting - an optimisation that should make something faster or smaller - a change to a data pipeline that should change counts or distributions Do not use it for refactors, API changes, or bugs a test can decide. Say so and skip the protocol. ## The protocol Follow the six steps in order. Do not skip ahead. Do not edit repository files before step 6. ### 1. Hypothesis One sentence: which cause produces which symptom, and what the change does about it. If you cannot write it in one sentence, you do not have a hypothesis yet. ### 2. Predictions, written before the first run Write a numbered list P1 to Pn. Every prediction needs a number and a threshold. "It gets better" is not a prediction. "Sharpe improves by at least 0.3 on the training window" is. The list must contain: - **A symptom check.** One prediction that the baseline reproduces the problem. If you cannot see the problem before the change, you cannot see it go away. - **A counter-prediction.** One thing the change will probably *not* achieve. Say the weak spot out loud before the run. - **A falsifier.** One outcome that, if it happens, means the hypothesis is wrong and the change must not be adopted. - **One prediction per independent symptom.** If the cause is real, several symptoms disappear together. Test each one separately. Then write the **decision rule**: which predictions must hold for the change to be adopted. Write it now, not after you see the numbers. ### 3. Patch, do not edit Write a script in a scratch directory, never in the repository. The script imports the real code and swaps out the one function, argument, or constant under test for the duration of the measurement. In Python use `unittest.mock.patch.object`. In other languages use whatever gives the same effect: an environment variable, a config override, a feature flag, a shadow copy of one file. Run the baseline and the patched version on the same inputs, same seeds, same window. Run the baseline first. If the full run is too slow, reduce it, and say in the report that you did. Use `templates/predict_first_template.py` as the starting point. ### 4. Measure Run the script. Collect the numbers the predictions ask for and nothing else. Do not start interpreting yet. ### 5. Verdict Fill the table, one row per prediction: ``` ID Prediction Measured Result P1 baseline drift > 100 % 4.2e13 holds P2 patched drift < 1 % -0.0023 holds P3 phase error still > 0.01 rad 0.09 rad holds P4 amplitude within 5 % +0.0003 holds VERDICT: confirmed, adopt the change ``` Rules for this step: - Predictions are frozen. Do not reword, loosen, or drop one after the run. - A failed prediction is reported as failed. Explaining it away is not allowed. "P1 fails, the gain was 0.02 not 0.3" is the whole sentence. - Apply the decision rule from step 2 as written. - Report the result to the user in full, including failures, before proposing anything. ### 6. Only now: act If the decision rule says adopt, propose the real change. Put the verdict table in the commit message or pull request so the evidence travels with the code. If the rule says do not adopt, say so, keep the scratch script as a record, and go back to step 1 with what you learned. ## Hard rules - No edits to repository files before the verdict. None. - Predictions before measurements, always, in writing, with numbers. - At least one counter-prediction and one falsifier per experiment. - Baseline and patch on identical inputs and seeds. - Frozen predictions. A failed one stays failed. - Reduced runs are declared as reduced. - The user sees the full table, not a summary of the good parts. ## Report format Use this structure when reporting back: ``` predict-first: VERDICT: Notes: ``` `templates/report.md` has the same skeleton for commit messages and pull requests.