# Eval Grader

> Grades and classifies evaluation batch results, applying exclusions, diagnosing failure modes, computing pass rates, and generating summary tables for papers.

- Skill: `samyakjhaveri/eval-grader` (Agent Skill)
- Install (CLI): `npx skillmds add samyakjhaveri/eval-grader`
- Raw SKILL.md: https://api.skillmd.com/api/skills/samyakjhaveri/eval-grader/raw
- Safety review: PASS (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Data & Analytics, Data Analysis
- Tags: Evaluation, Failure Classification, Grading, Markdown Table, Pass Rate, Statistics
- Author: SamyakJhaveri (https://skillmd.com/u/samyakjhaveri)
- Updated: 2026-08-22
- Page: https://skillmd.com/skills/samyakjhaveri/eval-grader

---


# Eval Result Grader

Structured workflow for grading and classifying evaluation results from batch runs.
Produces defensible statistics for paper reporting.

**Trigger:** `/eval-grader` or when classifying/summarizing eval batch results.

## When to use

- After `/post-eval` completes and you need to classify results
- Computing pass rates for paper tables
- Diagnosing systematic failure patterns across models
- Comparing two models' results on the same task set

## Arguments

- `<result-path>` — path to a result JSON or directory of results
- `--model <name>` — filter to specific model
- `--config <name>` — filter to specific configuration

## Grading Pipeline

### Step 1: Load Results

Read result files from the project's results directory. Each result should contain
at minimum: task identifier, model, configuration, and outcome status.

### Step 2: Apply Exclusions

Apply known-failing case exclusions. The exclusion list should be maintained in
`known-issues.md` or equivalent. Key rules:
- Exclude cases where the task itself is known to be broken
- Document the exclusion formula and denominator clearly
- Report excluded count alongside statistics

### Step 3: Classify Failure Modes

For non-passing results, classify into fine-grained categories relevant to the
project's domain. Build a failure taxonomy with:
- Category and subcategory
- Typical root cause
- Count and percentage of total failures

### Step 4: Compute Statistics

Report per model and configuration:
```
Pass rate:       PASS / N_valid x 100
Failure breakdown: <category>: X%, <category>: Y%
```

For papers: report both single-sample and multi-sample pass rates if applicable.

### Step 5: Generate Summary Table

Output a markdown table suitable for paper inclusion.

## Verification Gates

Before reporting any statistic:
1. Confirm exclusion list was applied
2. Confirm denominator matches expected task count
3. Cross-check at least 2-3 individual results manually
4. Flag anomalies (e.g., tasks that pass at baseline but fail with augmentation)