Eval Result Grader
Structured workflow for grading and classifying evaluation results from batch runs. Produces defensible statistics for paper reporting.
Trigger: /eval-grader or when classifying/summarizing eval batch results.
When to use
- After
/post-evalcompletes and you need to classify results - Computing pass rates for paper tables
- Diagnosing systematic failure patterns across models
- Comparing two models' results on the same task set
Arguments
<result-path>— path to a result JSON or directory of results--model <name>— filter to specific model--config <name>— filter to specific configuration
Grading Pipeline
Step 1: Load Results
Read result files from the project's results directory. Each result should contain at minimum: task identifier, model, configuration, and outcome status.
Step 2: Apply Exclusions
Apply known-failing case exclusions. The exclusion list should be maintained in
known-issues.md or equivalent. Key rules:
- Exclude cases where the task itself is known to be broken
- Document the exclusion formula and denominator clearly
- Report excluded count alongside statistics
Step 3: Classify Failure Modes
For non-passing results, classify into fine-grained categories relevant to the project's domain. Build a failure taxonomy with:
- Category and subcategory
- Typical root cause
- Count and percentage of total failures
Step 4: Compute Statistics
Report per model and configuration:
Pass rate: PASS / N_valid x 100
Failure breakdown: <category>: X%, <category>: Y%
For papers: report both single-sample and multi-sample pass rates if applicable.
Step 5: Generate Summary Table
Output a markdown table suitable for paper inclusion.
Verification Gates
Before reporting any statistic:
- Confirm exclusion list was applied
- Confirm denominator matches expected task count
- Cross-check at least 2-3 individual results manually
- Flag anomalies (e.g., tasks that pass at baseline but fail with augmentation)
