Tool Shadow Calibration

Tool Shadow Calibration

Use this skill before trusting a new Tool Shadow, after an authoritative or physical result contradicts it, and periodically as the design space or source tool changes.

A Tool Shadow is useful only when it improves decisions. Low mean error is not enough if the shadow ranks finalists incorrectly, misses unsafe cases, or fails to abstain outside its training envelope.

Calibration workflow

1. Freeze the shadow version, dataset version, feature contract, units, and authoritative evaluator version.
2. Maintain held-out tests that include ordinary cases, constraint-boundary cases, withheld geometry or input corners, and optimizer-generated challenge cases.
3. Compare shadow predictions with authoritative and physical results without overwriting any prior Evaluation.
4. Measure decision-aware metrics:
   • false-safe rate
   • top-k feasible recall
   • constraint-boundary accuracy
   • ranking regret
   • interval calibration
   • abstention and out-of-domain quality
   • authoritative-call reduction
   • first-finalist verification rate
   • disagreement between verified and observed outcomes
5. Select new authoritative evaluations using a mixture of:
   • high predicted value
   • high uncertainty
   • proximity to constraints
   • evaluator disagreement
   • diversity
   • known failure patterns
6. Append the results to a versioned dataset and retrain or recalibrate only through an auditable Run.
7. Compare the new shadow against the prior version on the same frozen suite.
8. Recommend promotion, restricted use, retraining, rollback, or retirement. Policy or a human authority makes the decision.

Optimizer-resistance

Assume an optimization agent will search for and amplify systematic shadow errors. Keep hard bounds and trust regions, penalize uncertainty, require authoritative checks near constraints, and add optimizer-discovered failures to a permanent challenge set.

Status discipline

Use separate labels:

• predicted: produced by a shadow
• verified: produced by an authoritative software tool
• observed: measured in the physical or production environment

A physical discrepancy may reveal a bad shadow, a bad authoritative model, incorrect conditions, manufacturing variation, or measurement error. Preserve all hypotheses until evidence distinguishes them.

Use https://github.com/closedloop-technologies/climbhill/blob/main/resources/strategies/tool-shadows/CALIBRATION.md for the full strategy and https://github.com/closedloop-technologies/climbhill/blob/main/resources/strategies/tool-shadows/CONTRACT.md for required provenance fields.