Output Grader

Output Grader Skill

Acts as an LLM judge to grade skill outputs against predefined assertions. Produces structured grading reports with concrete evidence for each pass or fail decision.

Workflow

Step 1: Load Assertions and Outputs

For each test case defined in <skill-dir>/evals/evals.json:

1. Read the assertions array.
2. Read the actual outputs from evals/workspace/quality-results/iteration-<N>/eval-<id>/with_skill/outputs/ and without_skill/outputs/.
3. Read the response.txt and any output files for each run.

Step 2: Evaluate Each Assertion

For each assertion, perform LLM-based evaluation against both with-skill and without-skill outputs:

Grading criteria:

• PASS: The assertion is demonstrably true. Must cite concrete evidence — a specific file name, a quoted string from the output, a measurable property (file size, count, format).
• FAIL: The assertion is false or cannot be verified from the available outputs.

Evidence requirements:

• Evidence must quote or reference specific output content (e.g., "Found chart.png (45KB) in with_skill/outputs/", or "Response line 12: 'X-axis: Q1, Q2, Q3, Q4'").
• "The output looks correct" is not evidence. Be specific.
• If the output is ambiguous, FAIL with evidence explaining the ambiguity.

Step 3: Review Assertion Quality

Before finalizing grades, evaluate the assertions themselves for quality issues:

• Always-pass assertion: The behavior is present even without the skill. Flag with assertion_quality: "always-pass". These assertions don't measure skill value and should be removed from future eval sets.
• Always-fail assertion: The behavior fails in both with-skill and without-skill configurations. Flag with assertion_quality: "always-fail". These indicate either broken assertions or genuinely impossible tasks.
• Unverifiable assertion: Cannot be checked from outputs (e.g., "The code is secure" without a security scanner). Flag with assertion_quality: "unverifiable".

Step 4: Produce Grading Files

For each eval case, write per-configuration grading.json files that match the state-protocol schema:

evals/workspace/quality-results/iteration-<N>/eval-<id>/with_skill/grading.json:

{
  "eval_id": "eval-1",
  "configuration": "with_skill",
  "assertion_results": [
    {
      "text": "Output includes a PNG image file",
      "passed": true,
      "evidence": "Found chart.png (45KB) in with_skill/outputs/",
      "assertion_quality": "valid"
    },
    {
      "text": "X-axis has month labels",
      "passed": true,
      "evidence": "Chart shows labels: Jan, Feb, Mar on x-axis (confirmed via chart metadata)",
      "assertion_quality": "valid"
    },
    {
      "text": "The chart uses optimal color scheme",
      "passed": false,
      "evidence": "Cannot objectively determine 'optimal' from output alone — subjective criterion",
      "assertion_quality": "unverifiable"
    }
  ],
  "summary": {"passed": 2, "failed": 1, "total": 3, "pass_rate": 0.66}
}

evals/workspace/quality-results/iteration-<N>/eval-<id>/without_skill/grading.json:

{
  "eval_id": "eval-1",
  "configuration": "without_skill",
  "assertion_results": [
    {
      "text": "Output includes a PNG image file",
      "passed": false,
      "evidence": "No PNG files found in without_skill/outputs/. Only response.txt present.",
      "assertion_quality": "valid"
    },
    {
      "text": "X-axis has month labels",
      "passed": false,
      "evidence": "No chart generated",
      "assertion_quality": "valid"
    },
    {
      "text": "The chart uses optimal color scheme",
      "passed": false,
      "evidence": "No chart generated",
      "assertion_quality": "unverifiable"
    }
  ],
  "summary": {"passed": 0, "failed": 3, "total": 3, "pass_rate": 0.0},
  "assertion_quality_flags": ["unverifiable"]
}

assertion_quality is recorded per-assertion in both files. The assertion_quality_flags summary array collects all flagged assertion qualities in the without_skill file.

Step 5: Summarize per Iteration

Optionally produce an aggregate summary across all eval cases in the iteration at evals/workspace/quality-results/iteration-<N>/grading-summary.json.

Gotchas

• Assertions that always pass in both configs: If an assertion passes regardless of skill presence, it provides no signal. These should be removed from the eval set — they inflate pass rates without measuring skill value.
• Assertions that always fail in both configs: If both with-skill and without-skill fail, the task may be impossible given the tools available, or the assertion is poorly specified. Do not penalize the skill for these — flag and exclude from pass rate.
• Don't give the benefit of the doubt: If evidence is missing or incomplete, the assertion FAILS. "Probably correct" is not a grade. Require explicit, quotable evidence.
• Subjective assertions are untestable: Assertions like "the output is good" or "the code is clean" cannot be objectively graded. Flag as unverifiable. Rewrite to be specific: "The output includes a docstring for every public function" or "The code passes ruff check with zero errors."
• With-skill and without-skill grading order matters: Grade both configurations independently. Don't let one configuration's result bias the other. Explicitly read each set of outputs before making judgments.
• File encoding issues: Some output files may be binary (images, PDFs). For these, grade based on file existence, size, and format metadata rather than content inspection.

Verification

Verify this skill produces correct output:

1. Create fixture outputs: with-skill has chart.png and a response mentioning axis labels; without-skill has only a text response.
2. Create assertions: "Output includes a PNG file" (should pass with-skill, fail without-skill), "Response mentions revenue" (should pass both), "Chart is interactive" (should fail both).
3. Run output-grader. Confirm grading.json has correct pass/fail per assertion per configuration.
4. Confirm "Response mentions revenue" is flagged as assertion_quality: "always-pass".
5. Confirm "Chart is interactive" is flagged as assertion_quality: "always-fail".
6. Confirm all PASS entries include specific evidence (not "looks good" or equivalent).