Hill Climb
The toolkit's metric-driven optimization loop. One number moves; everything else
stays fixed. Each iteration states one hypothesis, makes one change, runs the
correctness floor, re-measures, and either accepts the change or reverts it. The
ledger of what was tried and what failed ships with the code.
Sibling to objective-loop: that loop verifies boolean criteria and reschedules;
this loop optimizes a continuous metric against variance. Route here whenever the
goal is a number moving in a direction.
Phase 1: SPEC
Fill these fields from the request. Interview only for what is missing.
| Field |
Meaning |
Required |
Default |
| METRIC |
One number, with units and direction (lower or higher is better) |
yes |
— |
| MEASURE |
A deterministic command that prints that number, repeatable |
yes |
— |
| TARGET |
The value that ends the loop |
yes |
— |
| FLOOR |
Correctness gate command(s) that must exit 0 every iteration |
yes |
— |
| FIXTURE |
Dataset, workload, or input identity, pinned to a commit or checksum |
yes |
— |
| Variance tolerance |
Spread below which a delta means nothing |
no |
2x the baseline spread |
| Iteration budget |
Iterations before a forced stop |
no |
8 |
| Plateau threshold K |
Consecutive non-improving iterations that stop the loop |
no |
3 |
Rules:
- One METRIC per loop. Two numbers with a trade-off need one of them promoted to the FLOOR (for example: "p99 latency drops, memory stays under 500 MB").
- MEASURE prints the number and nothing that requires interpretation. Wrap noisy tools in a script that emits one value.
- A hill climb against a varying dataset measures nothing. Pin FIXTURE before Phase 2 — same input rows, same seed, same machine class, same warm/cold state.
- FLOOR is executed, not asserted. Name the command.
- A domain playbook fills this table fast:
references/domain-playbooks.md carries pre-filled blocks for frame rate, API latency, CI time, test runtime, bundle size, memory, token cost, and game-design quality.
- A judgment score can be the METRIC only under the frozen-rubric contract in that reference: rubric frozen at SPEC time, graded in a fresh context by an agent that did not author the change, wider accept threshold. Without a freezable rubric the request is not hill-climbable — that reference names where it goes instead.
Gate: all required fields hold concrete values. Proceed to Phase 2.
Phase 2: BASELINE
Run MEASURE N times (N ≥ 5, N ≥ 10 for wall-clock metrics) before changing any
code. Record every sample, the median, and the spread (max − min, or p95 − p5).
| Condition |
Action |
| Spread < target improvement |
Proceed. Set the variance tolerance from the spread. |
| Spread ≥ target improvement |
Stop. Report that the harness is too noisy to hill-climb on. |
A noisy-harness stop is a correct outcome, not a failure. Report the measured
spread, name the likely noise sources (shared CI runners, thermal throttling,
network calls, unpinned data, garbage-collection timing), and offer to stabilize
the harness first. Never proceed by averaging harder and hoping.
Gate: baseline median and spread written to the ledger. Proceed to Phase 3.
Phase 3: PROFILE
Locate the cost before changing anything. Guessing at hot spots is the dominant
failure mode of optimization work.
| Domain |
Tooling |
| Python CPU |
py-spy record, cProfile + snakeviz, pyinstrument |
| Python memory |
memray, tracemalloc |
| Go |
pprof (-cpuprofile, -memprofile), go test -bench -benchmem, benchstat |
| Browser runtime and frame rate |
Chrome DevTools performance trace; in this harness mcp__chrome-devtools__performance_start_trace, performance_stop_trace, performance_analyze_insight, take_heapsnapshot |
| Bundle size |
webpack-bundle-analyzer, rollup-plugin-visualizer, source-map-explorer |
| Test runtime |
pytest --durations=25, vitest --reporter=verbose, go test -json timings |
| CI wall-clock |
Per-job and per-step durations from the CI API; critical-path analysis across the job graph |
| Token cost |
Per-call token counts by prompt component; context-size attribution |
Write one hypothesis before the edit, in this shape: "X consumes N% of METRIC
because Y; doing Z should recover about M." Hypothesis first, edit second.
Gate: profile output captured, one hypothesis written to the ledger. Proceed to Phase 4.
Phase 4: ITERATE
Per iteration, in order:
- State one hypothesis (from Phase 3, or re-profiled after the last accept).
- Make one change. One change per iteration — bundled changes cannot be attributed.
- Run FLOOR. Red floor ends the iteration: revert, log, next.
- Run MEASURE the same N times as the baseline, same fixture, same conditions.
- Compare the new median to the current best.
| Floor |
Metric vs best |
Action |
| Green |
Improved beyond variance tolerance |
Accept. New best. Log. |
| Green |
Improved within variance tolerance |
Revert. Noise, not a win. Log as inconclusive. |
| Green |
Worse or unchanged |
Revert. Log the negative result — it is the valuable part. |
| Red |
Any |
Revert. Log the floor failure. |
Guardrails (hard rules). Never move the number by moving the goalposts:
- Never edit, shrink, reseed, or re-sample the FIXTURE.
- Never relax, skip, mark-xfail, or delete a FLOOR test.
- Never shrink the workload, lower the iteration count, or cut the input size.
- Never change MEASURE mid-loop; a changed measure invalidates the baseline and restarts the loop.
- Never cache away work the metric is supposed to include, or move it outside the measured region, unless the user accepts that as the intended change.
- Never accept on a single sample.
Hitting a guardrail is a stop-and-report event, not a judgment call. If the only
visible path to TARGET crosses one, stop and put the conflict to the user.
Gate: iteration logged with accept/revert and evidence. Loop until a Phase 5 stop condition fires.
Phase 5: LEDGER
.hillclimb/<slug>/ledger.md is append-only and written every iteration.
Entries are never edited retroactively. Template and field rules:
references/ledger.md.
Per iteration record: hypothesis, change summary (files touched), metric median
and spread, delta vs baseline, delta vs best, floor status, accepted or reverted,
and why.
The ledger ships with the code. It tells the next person what was tried and what
did not work, which is what stops the same dead end being re-walked. Keep
.hillclimb/ unstaged unless the user asks for the ledger in the repo.
Phase 6: STOP
| Condition |
Report |
| TARGET hit |
Baseline → final value, accepted changes in order with their deltas, floor green, ledger path |
| Iteration budget exhausted |
Best achieved value, remaining gap to TARGET, next hypothesis in the queue |
| Plateau: K consecutive non-improving iterations |
Best achieved value, the hot spots the profiler still shows, and what a next attempt would need |
| Noisy harness (Phase 2) |
Measured spread, noise sources, harness-stabilization proposal |
| Guardrail conflict (Phase 4) |
The guardrail, the change that would have crossed it, and the decision put to the user |
On plateau, name the shape of the next attempt honestly: a different algorithm,
a different data structure, an accepted architectural change, or "the remaining
cost is irreducible at this design." Do not silently keep grinding past K.
Error Handling
| Error |
Cause |
Solution |
| MEASURE prints varying numbers for identical code |
Unpinned fixture, shared machine, network in the measured path |
Return to Phase 2; pin the fixture or stabilize the harness before iterating |
| FLOOR was already red at baseline |
Pre-existing failure |
Stop. Fix correctness first — a hill climb over a red floor accepts nothing |
| Metric improves but the floor flakes intermittently |
Flaky test in FLOOR |
Treat as red; stabilize or replace that gate. Never downgrade a flaky gate to make an accept stick |
| Large win from one change looks too good |
Work was cached, skipped, or moved out of the measured region |
Verify the workload still executes: assert output equality against the baseline run before accepting |
| Ledger missing on resume |
.hillclimb/ removed mid-loop |
Re-run Phase 2 baseline; do not resume from memory |
Reference Loading Table
| Signal |
Load These Files |
Why |
| Writing or resuming the ledger |
references/ledger.md |
Append-only template, slug rules, resume protocol |
| Choosing a profiler, or reading its output |
references/profiling-tools.md |
Per-domain commands, output reading, common traps |
| Phase 1 in a named domain — frame rate, API latency, CI time, test runtime, bundle size, memory, token cost, game-design quality |
references/domain-playbooks.md |
Pre-filled METRIC/MEASURE/FIXTURE/FLOOR blocks and typical hot spots |
| The metric is a judgment score, or the request may not be measurable at all |
references/domain-playbooks.md |
Frozen-rubric contract and the not-hill-climbable boundary |
References
${CLAUDE_SKILL_DIR}/references/ledger.md — ledger template and resume protocol${CLAUDE_SKILL_DIR}/references/profiling-tools.md — per-domain profiling commands and traps${CLAUDE_SKILL_DIR}/references/domain-playbooks.md — pre-filled SPEC blocks per domain, rubric-metric contract, not-hill-climbable boundary
1---2name: hill-climb3description: Metric-driven optimization loop: baseline a number, profile the cost, then accept or revert one change per iteration on measured evidence until the target is hit or the curve plateaus. Use when the user says "make this faster", "reduce memory", "speed up CI", "get p99 under 200ms", "improve the frame rate", "cut the bundle size", "make the tests run quicker", or "reduce token cost". Requires one number, a repeatable command that prints it, and a fixed dataset; the loop stops and says so when the harness is too noisy to measure the improvement being chased.4---56# Hill Climb78The toolkit's metric-driven optimization loop. One number moves; everything else9stays fixed. Each iteration states one hypothesis, makes one change, runs the10correctness floor, re-measures, and either accepts the change or reverts it. The11ledger of what was tried and what failed ships with the code.1213Sibling to `objective-loop`: that loop verifies boolean criteria and reschedules;14this loop optimizes a continuous metric against variance. Route here whenever the15goal is a number moving in a direction.1617## Phase 1: SPEC1819Fill these fields from the request. Interview only for what is missing.2021| Field | Meaning | Required | Default |22|---|---|---|---|23| METRIC | One number, with units and direction (lower or higher is better) | yes | — |24| MEASURE | A deterministic command that prints that number, repeatable | yes | — |25| TARGET | The value that ends the loop | yes | — |26| FLOOR | Correctness gate command(s) that must exit 0 every iteration | yes | — |27| FIXTURE | Dataset, workload, or input identity, pinned to a commit or checksum | yes | — |28| Variance tolerance | Spread below which a delta means nothing | no | 2x the baseline spread |29| Iteration budget | Iterations before a forced stop | no | 8 |30| Plateau threshold K | Consecutive non-improving iterations that stop the loop | no | 3 |3132Rules:3334- One METRIC per loop. Two numbers with a trade-off need one of them promoted to the FLOOR (for example: "p99 latency drops, memory stays under 500 MB").35- MEASURE prints the number and nothing that requires interpretation. Wrap noisy tools in a script that emits one value.36- A hill climb against a varying dataset measures nothing. Pin FIXTURE before Phase 2 — same input rows, same seed, same machine class, same warm/cold state.37- FLOOR is executed, not asserted. Name the command.38- A domain playbook fills this table fast: `references/domain-playbooks.md` carries pre-filled blocks for frame rate, API latency, CI time, test runtime, bundle size, memory, token cost, and game-design quality.39- A judgment score can be the METRIC only under the frozen-rubric contract in that reference: rubric frozen at SPEC time, graded in a fresh context by an agent that did not author the change, wider accept threshold. Without a freezable rubric the request is not hill-climbable — that reference names where it goes instead.4041Gate: all required fields hold concrete values. Proceed to Phase 2.4243## Phase 2: BASELINE4445Run MEASURE N times (N ≥ 5, N ≥ 10 for wall-clock metrics) before changing any46code. Record every sample, the median, and the spread (max − min, or p95 − p5).4748| Condition | Action |49|---|---|50| Spread < target improvement | Proceed. Set the variance tolerance from the spread. |51| Spread ≥ target improvement | **Stop.** Report that the harness is too noisy to hill-climb on. |5253A noisy-harness stop is a correct outcome, not a failure. Report the measured54spread, name the likely noise sources (shared CI runners, thermal throttling,55network calls, unpinned data, garbage-collection timing), and offer to stabilize56the harness first. Never proceed by averaging harder and hoping.5758Gate: baseline median and spread written to the ledger. Proceed to Phase 3.5960## Phase 3: PROFILE6162Locate the cost before changing anything. Guessing at hot spots is the dominant63failure mode of optimization work.6465| Domain | Tooling |66|---|---|67| Python CPU | `py-spy record`, `cProfile` + `snakeviz`, `pyinstrument` |68| Python memory | `memray`, `tracemalloc` |69| Go | `pprof` (`-cpuprofile`, `-memprofile`), `go test -bench -benchmem`, `benchstat` |70| Browser runtime and frame rate | Chrome DevTools performance trace; in this harness `mcp__chrome-devtools__performance_start_trace`, `performance_stop_trace`, `performance_analyze_insight`, `take_heapsnapshot` |71| Bundle size | `webpack-bundle-analyzer`, `rollup-plugin-visualizer`, `source-map-explorer` |72| Test runtime | `pytest --durations=25`, `vitest --reporter=verbose`, `go test -json` timings |73| CI wall-clock | Per-job and per-step durations from the CI API; critical-path analysis across the job graph |74| Token cost | Per-call token counts by prompt component; context-size attribution |7576Write one hypothesis before the edit, in this shape: *"X consumes N% of METRIC77because Y; doing Z should recover about M."* Hypothesis first, edit second.7879Gate: profile output captured, one hypothesis written to the ledger. Proceed to Phase 4.8081## Phase 4: ITERATE8283Per iteration, in order:84851. State one hypothesis (from Phase 3, or re-profiled after the last accept).862. Make one change. One change per iteration — bundled changes cannot be attributed.873. Run FLOOR. Red floor ends the iteration: revert, log, next.884. Run MEASURE the same N times as the baseline, same fixture, same conditions.895. Compare the new median to the current best.9091| Floor | Metric vs best | Action |92|---|---|---|93| Green | Improved beyond variance tolerance | Accept. New best. Log. |94| Green | Improved within variance tolerance | Revert. Noise, not a win. Log as inconclusive. |95| Green | Worse or unchanged | Revert. Log the negative result — it is the valuable part. |96| Red | Any | Revert. Log the floor failure. |9798**Guardrails (hard rules).** Never move the number by moving the goalposts:99100- Never edit, shrink, reseed, or re-sample the FIXTURE.101- Never relax, skip, mark-xfail, or delete a FLOOR test.102- Never shrink the workload, lower the iteration count, or cut the input size.103- Never change MEASURE mid-loop; a changed measure invalidates the baseline and restarts the loop.104- Never cache away work the metric is supposed to include, or move it outside the measured region, unless the user accepts that as the intended change.105- Never accept on a single sample.106107Hitting a guardrail is a stop-and-report event, not a judgment call. If the only108visible path to TARGET crosses one, stop and put the conflict to the user.109110Gate: iteration logged with accept/revert and evidence. Loop until a Phase 5 stop condition fires.111112## Phase 5: LEDGER113114`.hillclimb/<slug>/ledger.md` is append-only and written every iteration.115Entries are never edited retroactively. Template and field rules:116`references/ledger.md`.117118Per iteration record: hypothesis, change summary (files touched), metric median119and spread, delta vs baseline, delta vs best, floor status, accepted or reverted,120and why.121122The ledger ships with the code. It tells the next person what was tried and what123did not work, which is what stops the same dead end being re-walked. Keep124`.hillclimb/` unstaged unless the user asks for the ledger in the repo.125126## Phase 6: STOP127128| Condition | Report |129|---|---|130| TARGET hit | Baseline → final value, accepted changes in order with their deltas, floor green, ledger path |131| Iteration budget exhausted | Best achieved value, remaining gap to TARGET, next hypothesis in the queue |132| Plateau: K consecutive non-improving iterations | Best achieved value, the hot spots the profiler still shows, and what a next attempt would need |133| Noisy harness (Phase 2) | Measured spread, noise sources, harness-stabilization proposal |134| Guardrail conflict (Phase 4) | The guardrail, the change that would have crossed it, and the decision put to the user |135136On plateau, name the shape of the next attempt honestly: a different algorithm,137a different data structure, an accepted architectural change, or "the remaining138cost is irreducible at this design." Do not silently keep grinding past K.139140## Error Handling141142| Error | Cause | Solution |143|---|---|---|144| MEASURE prints varying numbers for identical code | Unpinned fixture, shared machine, network in the measured path | Return to Phase 2; pin the fixture or stabilize the harness before iterating |145| FLOOR was already red at baseline | Pre-existing failure | Stop. Fix correctness first — a hill climb over a red floor accepts nothing |146| Metric improves but the floor flakes intermittently | Flaky test in FLOOR | Treat as red; stabilize or replace that gate. Never downgrade a flaky gate to make an accept stick |147| Large win from one change looks too good | Work was cached, skipped, or moved out of the measured region | Verify the workload still executes: assert output equality against the baseline run before accepting |148| Ledger missing on resume | `.hillclimb/` removed mid-loop | Re-run Phase 2 baseline; do not resume from memory |149150## Reference Loading Table151152| Signal | Load These Files | Why |153|---|---|---|154| Writing or resuming the ledger | `references/ledger.md` | Append-only template, slug rules, resume protocol |155| Choosing a profiler, or reading its output | `references/profiling-tools.md` | Per-domain commands, output reading, common traps |156| Phase 1 in a named domain — frame rate, API latency, CI time, test runtime, bundle size, memory, token cost, game-design quality | `references/domain-playbooks.md` | Pre-filled METRIC/MEASURE/FIXTURE/FLOOR blocks and typical hot spots |157| The metric is a judgment score, or the request may not be measurable at all | `references/domain-playbooks.md` | Frozen-rubric contract and the not-hill-climbable boundary |158159## References160161- `${CLAUDE_SKILL_DIR}/references/ledger.md` — ledger template and resume protocol162- `${CLAUDE_SKILL_DIR}/references/profiling-tools.md` — per-domain profiling commands and traps163- `${CLAUDE_SKILL_DIR}/references/domain-playbooks.md` — pre-filled SPEC blocks per domain, rubric-metric contract, not-hill-climbable boundary
notque