Semantic Consistency Auditor

Semantic Consistency Auditor

Goal

Detect semantic mismatches that pass schema checks but break assessment validity or grading fairness.

Inputs

• mission.json
• statement.md
• rubric.md
• teacher-notes.md
• Optional reference/ and tests/

Produce

Write semantic-consistency-report.json with:

• overall_status (pass|review|fail)
• findings (id, severity, artifact, evidence, remediation)
• coverage_matrix (learning outcomes x rubric criteria x tasks/tests)
• leakage_risks (teaching/exam mismatch, hidden prerequisites, rubric blind spots)
• fairness_risks (ambiguous wording, non-deterministic grading, time-budget mismatch)

Mandatory checks

• Every assessed task maps to at least one declared learning outcome.
• Every rubric criterion maps to observable evidence in statement or expected outputs.
• Teacher notes do not introduce criteria absent from rubric.
• Tests (if present) verify intended competencies and do not overfit incidental details.
• Time/difficulty estimate is consistent with mission constraints.

Severity policy

• fail: invalidates grading integrity (missing alignment, contradictory criteria, impossible constraints).
• review: ambiguous but potentially salvageable.
• pass: no blocking inconsistency detected.

Rules

• Prioritize evidence-based findings over style feedback.
• Report concrete remediation steps, not generic advice.
• Never mutate source artifacts; emit audit only.