Harness Stripping
Every harness component was added to compensate for a specific model failure. Models improve. Components don't retire themselves. The scaffolding that saved you on Sonnet 4.5 may be dead weight — or actively harmful — on Opus 4.6. Strip it deliberately, one piece at a time, and let evals tell you what still earns its keep.
Inspired by Prithvi's March 2026 harness post on evaluator-generator separation and the general "re-test your assumptions each model bump" discipline.
When to apply
- A model upgrade just landed and your harness was tuned for the previous generation.
- A component's justification is "we added this because the model used to do X" — and you haven't checked whether it still does X.
- The harness has accreted over months and nobody remembers what half the machinery is for.
- Cost or latency is climbing and you suspect redundant belt-and-suspenders layers.
Procedure
Inventory the components. List every distinct piece of scaffolding: prompt sections, tool wrappers, post-hoc validators, retry loops, evaluator personas, structured-output enforcers, sandbox rules. One row per component. Note the failure mode each was added to prevent.
Rank by suspicion. Put the components most likely to be obsolete at the top: anything added before the last two model bumps, anything targeting a failure mode you haven't seen recently, anything whose original justification is now folklore.
Pick a baseline eval. You need a repeatable metric before you touch anything. Reuse an existing eval set if you have one; otherwise pick 20–60 tasks representative of production work. Record baseline score, cost, and wall-clock.
Strip one component. Only one. Comment it out or gate it behind a flag — don't delete yet. Re-run the eval.
Compare against baseline.
- Score within noise, cost/latency down → the component is dead weight. Delete.
- Score drops measurably → the component still earns its complexity. Restore and note what failure mode returned.
- Score improves → the component was actively harmful. Delete and investigate why (often: over-constraining a now-capable model).
Commit the delta. Land the strip (or the restore-with-notes) as its own commit. Do not batch multiple strips into one change — you lose the ability to attribute the score movement.
Repeat for the next component. Re-establish baseline from the new state each round, not the original. Compounding strips have compounding effects.
Anti-patterns
- Stripping two components at once — you can't tell which one mattered. Halve the signal, double the confusion.
- Skipping the eval "because it's obviously safe to remove" — the harness accreted for reasons. Some are still real. Measure.
- Deleting instead of gating on the first pass — you will want to A/B mid-review. Flag first, delete after the eval confirms.
- Trusting anecdotes over the eval — "it feels better without it" is how load-bearing components get removed. If the eval doesn't show it, it isn't there.
- Stripping components that guard safety, sandboxing, or cost caps — those aren't compensating for model weakness. Leave them.
- Doing this on prod traffic — run against an eval set, not real users. The failure modes you're re-probing are exactly the ones that hurt users.
What to strip first
Highest yield in practice:
- Structured-output enforcers layered on top of models that now emit valid JSON natively.
- Multi-step "plan then execute" wrappers on tasks the model now one-shots.
- Retry loops around tool calls that no longer flake.
- Evaluator personas whose critiques the generator now anticipates on its own.
- Verbose "remember to do X" prompt sections where X is now default behavior.
When NOT to apply
Don't strip mid-project on a live long-running run — you'll perturb sessions in flight. Do it between projects, or on a forked branch. Also skip if you don't have an eval you trust; stripping without measurement is guessing.
Related
- [[shift-notes]] — record which components were stripped and when, so the next audit doesn't re-strip and re-restore the same piece.
- [[adversarial-verify]] — the evaluator-generator pattern that may itself be a strip candidate on newer models.
- [[broken-window-check]] — if you strip a component and the eval regresses in a specific way, that's your new broken window to hunt.
1---2name: harness-stripping3description: Systematically remove one harness component at a time and measure impact, killing scaffolding that no longer earns its complexity.4---56# Harness Stripping78Every harness component was added to compensate for a specific model failure. Models improve. Components don't retire themselves. The scaffolding that saved you on Sonnet 4.5 may be dead weight — or actively harmful — on Opus 4.6. Strip it deliberately, one piece at a time, and let evals tell you what still earns its keep.910Inspired by Prithvi's March 2026 harness post on evaluator-generator separation and the general "re-test your assumptions each model bump" discipline.1112## When to apply1314- A model upgrade just landed and your harness was tuned for the previous generation.15- A component's justification is "we added this because the model used to do X" — and you haven't checked whether it still does X.16- The harness has accreted over months and nobody remembers what half the machinery is for.17- Cost or latency is climbing and you suspect redundant belt-and-suspenders layers.1819## Procedure20211. **Inventory the components.** List every distinct piece of scaffolding: prompt sections, tool wrappers, post-hoc validators, retry loops, evaluator personas, structured-output enforcers, sandbox rules. One row per component. Note the failure mode each was added to prevent.22232. **Rank by suspicion.** Put the components most likely to be obsolete at the top: anything added before the last two model bumps, anything targeting a failure mode you haven't seen recently, anything whose original justification is now folklore.24253. **Pick a baseline eval.** You need a repeatable metric before you touch anything. Reuse an existing eval set if you have one; otherwise pick 20–60 tasks representative of production work. Record baseline score, cost, and wall-clock.26274. **Strip one component.** Only one. Comment it out or gate it behind a flag — don't delete yet. Re-run the eval.28295. **Compare against baseline.**30 - Score within noise, cost/latency down → the component is dead weight. Delete.31 - Score drops measurably → the component still earns its complexity. Restore and note *what failure mode* returned.32 - Score *improves* → the component was actively harmful. Delete and investigate why (often: over-constraining a now-capable model).33346. **Commit the delta.** Land the strip (or the restore-with-notes) as its own commit. Do not batch multiple strips into one change — you lose the ability to attribute the score movement.35367. **Repeat for the next component.** Re-establish baseline from the *new* state each round, not the original. Compounding strips have compounding effects.3738## Anti-patterns3940- **Stripping two components at once** — you can't tell which one mattered. Halve the signal, double the confusion.41- **Skipping the eval "because it's obviously safe to remove"** — the harness accreted for reasons. Some are still real. Measure.42- **Deleting instead of gating on the first pass** — you will want to A/B mid-review. Flag first, delete after the eval confirms.43- **Trusting anecdotes over the eval** — "it feels better without it" is how load-bearing components get removed. If the eval doesn't show it, it isn't there.44- **Stripping components that guard safety, sandboxing, or cost caps** — those aren't compensating for model weakness. Leave them.45- **Doing this on prod traffic** — run against an eval set, not real users. The failure modes you're re-probing are exactly the ones that hurt users.4647## What to strip first4849Highest yield in practice:5051- Structured-output enforcers layered on top of models that now emit valid JSON natively.52- Multi-step "plan then execute" wrappers on tasks the model now one-shots.53- Retry loops around tool calls that no longer flake.54- Evaluator personas whose critiques the generator now anticipates on its own.55- Verbose "remember to do X" prompt sections where X is now default behavior.5657## When NOT to apply5859Don't strip mid-project on a live long-running run — you'll perturb sessions in flight. Do it between projects, or on a forked branch. Also skip if you don't have an eval you trust; stripping without measurement is guessing.6061## Related6263- [[shift-notes]] — record which components were stripped and when, so the next audit doesn't re-strip and re-restore the same piece.64- [[adversarial-verify]] — the evaluator-generator pattern that may itself be a strip candidate on newer models.65- [[broken-window-check]] — if you strip a component and the eval regresses in a specific way, that's your new broken window to hunt.
archive228