# Calibration

> Predict-then-score protocol. Use at the START of any non-trivial task (multi-file change, debugging session, design decision, anything you could be wrong about) to log a falsifiable prediction, and at the END to score it. Also use when the user asks "how confident are you", when choosing between approaches, or when /calibrate reports overconfidence in a category. This protocol IS the harness's verified self-awareness; skipping it makes your judgement unmeasurable.

- Skill: `ghostlygawd/calibration` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ghostlygawd/calibration`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ghostlygawd/calibration/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: GhostlyGawd (https://skillmd.com/u/ghostlygawd)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ghostlygawd/calibration

---


# Calibration

Self-awareness you can't audit is vibes. This protocol replaces it with a ledger.

## Before acting

State, concretely, what success looks like — then log it:

    harness predict --task "fix flaky auth test" \
      --expect "root cause is async teardown; <=2 files touched; suite green in one run" \
      --confidence 0.7 --category debugging

A good `--expect` is one reality can contradict: file counts, pass/fail, the
named root cause, "user accepts without edits". "I will do a good job" is not
a prediction, it's a mood.

Beware the self-confirming `--expect`: a clause that asserts your OWN output is
correct ("surfaces the KNOWN X", "matches the map I built") makes a "hit" prove
only that the code reproduced your assumption — not that the assumption was right.
Tie at least one clause to something you did NOT author: user-confirmed intent,
external ground truth, an independent check. (2026-06-21: scored a "hit" on
"overlap detection surfaces the then-known brainstorm+Huashu overlap" — the "known"
was my own hand-label, later admitted an artifact; the hit certified a mislabel.)

Confidence honesty: 0.9 means 9-in-10. If your `high` bucket hits at 60%,
you are not unlucky, you are lying to yourself with extra steps.

## After acting

    harness outcome <id> --result hit|miss --notes "actual cause was fixture ordering"

Score EVERY prediction, especially misses — misses are where the information
is. The SessionStart banner shows your unscored count; keep it near zero.

## Don't self-grade the user's acceptance

If a prediction's success clause includes the user accepting something ("user accepts",
"<=2 rounds of correction", "no edits requested"), you may NOT score it until the user has
actually responded to that thing. Self-declared acceptance is not an outcome — it is you
grading from your own seat. Until the user reacts, the prediction stays `pending` (open debt),
never a self-awarded `hit`.

Corollary: a prediction that bundles a mechanical clause (the loop runs) with an acceptance
clause (the user accepts the design) can be a mechanical HIT and an acceptance MISS at once.
Log them as TWO predictions up front (the `outcome` CLI scores one result per id), or wait for the user — a green mechanical result must not let you round
the acceptance clause up to "hit".

provenance: session 3772bd2d, 2026-06-20 — scored prediction f156c20b a "hit" (its clause:
"user accepts ... with <=2 rounds of correction") minutes BEFORE the user delivered a
fundamental, design-invalidating correction. Acceptance was self-graded from inside Claude's own
frame; the real outcome was a miss on that clause.

## A load-bearing prediction gates shipping

If a prediction underwrites the DELIVERABLE — its core behavioral claim, the
thing the user actually asked for — then while it is still `pending` or scored
`miss`, the work is NOT done and must NOT merge. Green gates (tests, lint, a
passing auditor, an eval run) verify the ARTIFACTS; they do not verify the CLAIM.
Resolve it by exercising the REAL path end-to-end — the actual tool / trigger /
environment, never a proxy — then score it before declaring done or merging.
Shipping on a self-flagged "unverified" load-bearing claim is how a green PR
delivers the opposite of the goal. (2026-06-20: merged a "rides into worktrees"
change while its own prediction 55b1735b said that exact behavior was unverified;
post-merge it scored a MISS and the whole goal had to be re-delivered.)

## Reading your own stats

`harness stats` reports two lenses on the same scored log — confidence buckets
(claimed vs. actual hit rate, auto-flagged on large drift) and a per-category
hit-rate breakdown (no flag; it always prints both):

- **Overconfident in a category** → before acting in that category, list two
  ways you could be wrong and check one. Note persistent drift in
  memory/calibration/notes.md with a date.
- **Underconfident** → stop hedging and over-asking; act, then verify.
- **Misses clustering on one category** → that's a missing skill or hook.
  Route it (skill: routing-learnings).

## Wrong-track detection in flight

While working, your prediction is the tripwire: the moment observed reality
diverges from `--expect` (extra files ballooning, new failure class, the
"quick fix" sprouting branches), stop and re-plan instead of pushing through.
That divergence-noticing reflex is the intuition this harness trains.