Eval

Measure whether a change to AI behavior actually improved things, the way a code change is measured: binary quality dimensions, a grid-sampled synthetic eval set with negative controls and a generated response side, a hand-labeled answer key, a judge written like code, then judge calibration with toolkit/calibrate.py (TP/TN/FP/FN counts, TPR, TNR, accuracy, precision, and a harsh/lenient bias reading) before gating on thresholds set before the run. Use when the user changed AI behavior (prompt wording, a skill description or body, an agent loop, a RAG pipeline) and needs to know whether it helped, or asks to "build an eval set", "calibrate a judge", or "evaluate the prompt". Not the harness command `claude plugin eval`, and not a substitute for ordinary code tests: those check deterministic code, this measures judgment behavior.

ariadoss Updated 12 repo stars

File contents

ariadoss/superskills commit 8adc754fb0

Frequently asked questions

npx get-skill add ariadoss/eval