Judge Evaluation

Use when changing or evaluating judge prompts, scoring thresholds, usage floors, or judgment eval corpora.

kunchenguid Updated 8.9k repo stars
Compact adviser status line: "Compact adviser: work appears completed or recorded. Run /compact to save tokens." shown above a terminal prompt

File contents

  • Judge design rule, measured: two one-sentence atomic questions composed in code beat any single question that folds two judgments together (TypeSafe's own guidance). Hill-climb prompt changes with eval/tools/earn.py (paired bootstrap) and read the usage-floor ladder with eval/tools/schedule.py; a clause stays only if it earns its place.
  • Judgment eval harness: packages/pi-extension/eval/. Session transcripts, labels, worksheets, and results stay in gitignored eval/local/. See that README for the corpus contract and label rubric.

kunchenguid/compact-adviser commit 56e4de6d9f

Frequently asked questions

npx get-skill add kunchenguid/judge-evaluation