Consult A Decision Oracle

Add an external probabilistic classifier to a decision path without letting it take the path over. Covers finding the externally-graded rows you are already logging, measuring the confidence separation between the overrides it got right and the ones it got wrong, choosing an operating point your data licenses, failing open at the call site, and proving the whole arrangement offline with no API key. Applies to any service returning a score you can order — TypeSafe AI's System One models (Jev) are the worked example. Most of the method transfers to a local model or a second heuristic, with the parts that do not called out where they arise. Use when a rule-based path is wrong often enough to hurt, when someone proposes replacing a heuristic with a model, when a confidence threshold needs a defensible value, or when an oracle already in production has never been graded.

pjt222 Updated 44 repo stars

File contents

pjt222/agent-almanac commit 2afb085843

Frequently asked questions

npx get-skill add pjt222/consult-a-decision-oracle