# Evaluation And Quality Harness

> Measures exactitude, quality, and stability of runs produced by the teaching-agent system using programmatic checks, rubric-based grading, and benchmark cases. Use when validating workflows, benchmarking outputs, or comparing runs.

- Skill: `alainlebret/evaluation-and-quality-harness` (Agent Skill)
- Install (CLI): `npx skillmds@latest add alainlebret/evaluation-and-quality-harness`
- Raw SKILL.md: https://api.skillmd.com/api/skills/alainlebret/evaluation-and-quality-harness/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: alainlebret (https://skillmd.com/u/alainlebret)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/alainlebret/evaluation-and-quality-harness

---


# Evaluation and Quality Harness

## Goal

Systematically verify that a workflow run produced the expected artefacts, that those artefacts meet the quality criteria defined in the shared policies, and that results are stable across multiple runs (no random drift).

## When to use

- After completing a workflow run to certify it before publication or student distribution.
- When comparing two versions of the same course or exam to measure improvement.
- When benchmarking correction stability (same submission graded twice should yield the same score).

## Inputs

- `run-manifest.json` — list of produced artefacts and run metadata
- `quality-report.json` — programmatic gate results from `check_quality_gate.py`
- `language-report.json` — language consistency results from `check_language_policy.py`
- `mission.json` — source of truth for expected quality level and constraints
- `rubric.md` — used to verify grading reproducibility
- `shared/quality-metrics/quality-metrics.md` — the quality dimensions to evaluate

## Output contract

| File | Description |
|---|---|
| `benchmark/harness-report.md` | Human-readable report: each quality dimension with a pass/review/fail verdict and evidence |
| `benchmark/benchmark-cases.json` | List of benchmark cases run, inputs used, scores obtained, stability delta |
| `benchmark/stability-delta.json` | If run twice: score difference per criterion; acceptable delta ≤ 0.5 points |

## Quality dimensions evaluated

For each dimension, produce a verdict (`pass` / `review` / `fail`) with evidence:

| Dimension | What to check |
|---|---|
| **Specification coverage** | Are all `learning_outcomes` from `mission.json` covered by at least one rubric criterion? |
| **Scientific exactitude** | Do technical claims in `statement.md` and `teacher-notes.md` match established facts? |
| **Internal coherence** | Does the rubric total match the stated maximum? Do criteria refer to tasks that appear in the statement? |
| **Level adequacy** | Is the expected cognitive load consistent with the academic level in `mission.json`? |
| **Workload realism** | Can a student realistically complete all tasks in the stated `duration`? |
| **Evaluability** | Can every rubric criterion be applied reproducibly without subjective interpretation? |
| **Stability** | Re-running the correction on the same submission yields scores within the acceptable delta. |

## Rules

- Never invent evidence. If a dimension cannot be evaluated without executing code, state that explicitly and mark it `review`.
- Always cite the specific artefact line or section that supports each verdict.
- If `overall_status` is not `pass` after all checks, emit a prioritised list of fixes.
- Distinguish programmatic failures (wrong output, missing file) from judgement failures (vague criterion).