tomevault-io evalyn-calibrate
Pre-flight
- Verify evaluation runs exist:
evalyn list-runs --limit 1
If no runs: "You need evaluation results first. Invoke evalyn-eval."
- Identify calibration targets:
evalyn analyze --run <latest-run-id>
Look for subjective metrics (type [llm]) with pass rates below 90%. Only subjective metrics can be calibrated - objective metrics are deterministic.
Step 1: Annotate Results
Run interactive annotation with per-metric mode for targeted feedback:
evalyn annotate --run-id <latest-run-id> --dataset <dataset-path> --per-metric
This is an interactive terminal session. The user will:
- See each item's input, output, and LLM judge results
- Agree or disagree with each metric's judgment
- Commands:
[y]es/pass,[n]o/fail,[s]kip,[v]iew full,[q]uit
Aim for 20-30+ annotations. Focus on disagreements. Annotations save immediately - quit and resume anytime.
Step 2: Run Calibration
Start with the basic optimizer (fast, single-shot analysis):
evalyn calibrate --metric-id <target-metric> --annotations <annotations-dir>
The --annotations flag points to the directory containing annotation files (created by evalyn annotate).
The output shows alignment metrics:
- Accuracy: overall agreement rate
- F1 Score: balanced precision/recall
- Cohen's Kappa: agreement adjusted for chance
Optimizer comparison
| Optimizer | Flag | Speed | Best For |
|---|---|---|---|
basic |
--optimizer basic |
Fast (1 API call) | First pass, small annotation sets |
ape |
--optimizer ape |
Medium | Exploring many prompt variants |
opro |
--optimizer opro |
Medium | Iterative refinement |
gepa-native |
--optimizer gepa-native |
Slow | Best quality, built-in token tracking |
gepa |
--optimizer gepa |
Slow | Evolutionary (requires pip install gepa) |
evoprompt |
--optimizer evoprompt |
Medium | Evolutionary mutation/crossover of prompts |
textgrad |
--optimizer textgrad |
Medium | Critique-revise gradient descent on text |
miprov2 |
--optimizer miprov2 |
Medium | Instruction + few-shot demo co-optimization |
promptbreeder |
--optimizer promptbreeder |
Slow | Self-referential evolutionary prompt search |
Optimizer-specific settings are available as CLI flags (e.g., --opro-iterations, --ape-candidates, --gepa-task-lm, --gepa-reflection-lm, --gepa-max-calls).
If basic doesn't improve alignment enough:
evalyn calibrate --metric-id <target-metric> --annotations <annotations-dir> --optimizer gepa
Step 3: Re-evaluate with Calibrated Judges
evalyn run-eval --dataset <dataset-path> --use-calibrated
The --use-calibrated flag loads optimized prompts from the calibrations/ directory.
Step 4: Compare Results
evalyn compare --run1 <original-run-id> --run2 <calibrated-run-id>
Check whether calibrated pass rates better match your expectations.
Step 5: Assess and Iterate
View calibration history:
evalyn list-calibrations
If alignment improved: calibration successful. The judges now match your expectations.
If alignment is still poor:
- Try a different optimizer (
gepafor maximum quality) - Add more annotations (more data = better calibration)
- Review the metric's rubric in the metrics JSON file - it may need manual editing
- Run
evalyn cluster-misalignments --run-id <run-id>to see patterns in disagreements
No Automatic Next Step
This is the terminal skill. The user decides:
- Iterate: add more annotations and re-calibrate
- Fix agent: if judges are correct and agent has real issues
- Expand testing:
evalyn simulate --dataset <path> --modes similar,outlierthen re-evaluate - Ship: export final report with
evalyn export --run <id> --format html
Source: shihongDev/evalyn — distributed by TomeVault.