# Tool Shadow Calibration

> Evaluate, challenge, calibrate, and improve a Tool Shadow using authoritative results, physical outcomes, active learning, and decision-aware metrics rather than average prediction error alone.

- Skill: `closedloop-technologies/tool-shadow-calibration` (Agent Skill)
- Install (CLI): `npx skillmds@latest add closedloop-technologies/tool-shadow-calibration`
- Raw SKILL.md: https://api.skillmd.com/api/skills/closedloop-technologies/tool-shadow-calibration/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: closedloop-technologies (https://skillmd.com/u/closedloop-technologies)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/closedloop-technologies/tool-shadow-calibration

---


# Tool Shadow Calibration

Use this skill before trusting a new Tool Shadow, after an authoritative or
physical result contradicts it, and periodically as the design space or source
tool changes.

A Tool Shadow is useful only when it improves decisions. Low mean error is not
enough if the shadow ranks finalists incorrectly, misses unsafe cases, or fails
to abstain outside its training envelope.

## Calibration workflow

1. Freeze the shadow version, dataset version, feature contract, units, and
   authoritative evaluator version.
2. Maintain held-out tests that include ordinary cases, constraint-boundary
   cases, withheld geometry or input corners, and optimizer-generated challenge
   cases.
3. Compare shadow predictions with authoritative and physical results without
   overwriting any prior Evaluation.
4. Measure decision-aware metrics:
   - false-safe rate
   - top-k feasible recall
   - constraint-boundary accuracy
   - ranking regret
   - interval calibration
   - abstention and out-of-domain quality
   - authoritative-call reduction
   - first-finalist verification rate
   - disagreement between verified and observed outcomes
5. Select new authoritative evaluations using a mixture of:
   - high predicted value
   - high uncertainty
   - proximity to constraints
   - evaluator disagreement
   - diversity
   - known failure patterns
6. Append the results to a versioned dataset and retrain or recalibrate only
   through an auditable Run.
7. Compare the new shadow against the prior version on the same frozen suite.
8. Recommend promotion, restricted use, retraining, rollback, or retirement.
   Policy or a human authority makes the decision.

## Optimizer-resistance

Assume an optimization agent will search for and amplify systematic shadow
errors. Keep hard bounds and trust regions, penalize uncertainty, require
authoritative checks near constraints, and add optimizer-discovered failures to
a permanent challenge set.

## Status discipline

Use separate labels:

- `predicted`: produced by a shadow
- `verified`: produced by an authoritative software tool
- `observed`: measured in the physical or production environment

A physical discrepancy may reveal a bad shadow, a bad authoritative model,
incorrect conditions, manufacturing variation, or measurement error. Preserve
all hypotheses until evidence distinguishes them.

Use
`https://github.com/closedloop-technologies/climbhill/blob/main/resources/strategies/tool-shadows/CALIBRATION.md` for the full
strategy and
`https://github.com/closedloop-technologies/climbhill/blob/main/resources/strategies/tool-shadows/CONTRACT.md` for required
provenance fields.