alainlebret Evaluation and Quality Harness
Goal
Systematically verify that a workflow run produced the expected artefacts, that those artefacts meet the quality criteria defined in the shared policies, and that results are stable across multiple runs (no random drift).
When to use
- After completing a workflow run to certify it before publication or student distribution.
- When comparing two versions of the same course or exam to measure improvement.
- When benchmarking correction stability (same submission graded twice should yield the same score).
Inputs
run-manifest.json— list of produced artefacts and run metadataquality-report.json— programmatic gate results fromcheck_quality_gate.pylanguage-report.json— language consistency results fromcheck_language_policy.pymission.json— source of truth for expected quality level and constraintsrubric.md— used to verify grading reproducibilityshared/quality-metrics/quality-metrics.md— the quality dimensions to evaluate
Output contract
| File | Description |
|---|---|
benchmark/harness-report.md |
Human-readable report: each quality dimension with a pass/review/fail verdict and evidence |
benchmark/benchmark-cases.json |
List of benchmark cases run, inputs used, scores obtained, stability delta |
benchmark/stability-delta.json |
If run twice: score difference per criterion; acceptable delta ≤ 0.5 points |
Quality dimensions evaluated
For each dimension, produce a verdict (pass / review / fail) with evidence:
| Dimension | What to check |
|---|---|
| Specification coverage | Are all learning_outcomes from mission.json covered by at least one rubric criterion? |
| Scientific exactitude | Do technical claims in statement.md and teacher-notes.md match established facts? |
| Internal coherence | Does the rubric total match the stated maximum? Do criteria refer to tasks that appear in the statement? |
| Level adequacy | Is the expected cognitive load consistent with the academic level in mission.json? |
| Workload realism | Can a student realistically complete all tasks in the stated duration? |
| Evaluability | Can every rubric criterion be applied reproducibly without subjective interpretation? |
| Stability | Re-running the correction on the same submission yields scores within the acceptable delta. |
Rules
- Never invent evidence. If a dimension cannot be evaluated without executing code, state that explicitly and mark it
review. - Always cite the specific artefact line or section that supports each verdict.
- If
overall_statusis notpassafter all checks, emit a prioritised list of fixes. - Distinguish programmatic failures (wrong output, missing file) from judgement failures (vague criterion).