tomevault-io Judge
Overview
Evaluate agent task outputs against a three-dimension rubric and produce structured verdict records. The judge operates as a quality gate at the task completion boundary, scoring outputs on Semantic accuracy, Pragmatic usefulness, and Syntactic consistency.
Rubric: Three KLS Dimensions
The rubric reuses three dimensions from the KLS (Krogstie-Lindland-Sindre) quality
framework defined in disciplined-quality-evaluation:
| Dimension | Question | Criteria |
|---|---|---|
| Semantic | Does it accurately represent the domain? | Factual correctness, domain terminology, no contradictions |
| Pragmatic | Does it enable the intended decisions/actions? | Actionable, useful, addresses the task goal |
| Syntactic | Is it internally consistent and well-structured? | Format compliance, structural completeness, no broken references |
Scoring
| Score | Meaning |
|---|---|
| 1 | Poor -- major issues, blocks use |
| 2 | Below Standard -- significant gaps |
| 3 | Adequate -- meets minimum bar |
| 4 | Good -- clear, useful, few issues |
| 5 | Excellent -- exemplary, no issues |
Verdict Thresholds
| Condition | Verdict |
|---|---|
| All dimensions >= 3 AND average >= 3.5 | accept |
| Any dimension < 3 OR average < 3.5, but all >= 2 | improve |
| Any dimension < 2 | reject |
| Models disagree on accept/reject | escalate |
Verdict Format
Every judge evaluation produces a JSON verdict record. See automation/judge/verdict-schema.json
for the full schema. Minimal structure:
{
"task_id": "issue-18",
"model": "opencode/gpt-5-nano",
"mode": "quick",
"verdict": "accept",
"scores": {
"semantic": 4,
"pragmatic": 4,
"syntactic": 5
},
"average": 4.33,
"reasoning": "Brief justification for scores",
"improvements": [],
"timestamp": "2026-02-17T14:30:00Z"
}
Output MUST be valid JSON only -- no markdown fencing, no preamble, no trailing text.
Judge Modes
Quick Judge
- Model: opencode/gpt-5-nano (fast, low latency)
- Purpose: Rapid pass/fail screening at task completion
- Timeout: 30 seconds
- Prompt template: references/prompt-quick.md
- Output: Verdict JSON with scores and one-line reasoning
Deep Judge
- Model: opencode/kimi-k2.5-free (thorough, reasoning-capable)
- Purpose: Detailed evaluation with improvement suggestions
- Timeout: 60 seconds
- Prompt template: references/prompt-deep.md
- Output: Verdict JSON with scores, reasoning chain, and improvement list
Tiebreaker
- Model: opencode/gpt-5.1-codex-mini (independent perspective, different provider)
- Purpose: Resolve accept/reject disagreement between quick and deep judges
- Timeout: 45 seconds
- Prompt template: Uses deep prompt with prior verdicts appended
- Trigger: Quick and deep verdicts disagree on accept vs reject/improve
Multi-Iteration Protocol
1. Run quick judge
|
+-- accept --> DONE (accept)
+-- reject --> DONE (reject, log improvements)
+-- improve --> Run deep judge
|
+-- accept --> DONE (accept)
+-- reject --> DONE (reject, log improvements)
+-- improve --> Human fallback
When quick and deep disagree on accept vs reject:
Quick: accept + Deep: reject --> Run tiebreaker
Quick: reject + Deep: accept --> Run tiebreaker
Tiebreaker verdict is final (no further iteration)
Maximum 3 model calls before human fallback.
Human Fallback Criteria
Escalate to human review when:
- Three model calls completed without consensus
- Any model returns invalid JSON after retry
- Any model times out after retry
- Verdict is "improve" from both quick and deep judges (ambiguous quality)
- Task involves security-sensitive or compliance-critical content
Integration Points
Runner Script
The judge is invoked by automation/judge/run-judge.sh (v2), which:
- Writes prompts to temp files (eliminates shell escaping issues with file content)
- Calls opencode with
--file <tempfile>for reliable prompt delivery - Parses verdict JSON from opencode's JSON event stream
- Implements the multi-iteration protocol (quick -> deep -> tiebreaker)
- Optionally enriches verdict logging with terraphim-cli term matching
Verdict Logging
All verdicts are appended to automation/judge/verdicts.jsonl as one JSON object per line.
This enables:
- Audit trail of all quality evaluations
- Trend analysis across tasks
- Model performance comparison
Pre-Push Hook
The judge can be invoked as a pre-push hook via automation/judge/pre-push-judge.sh,
blocking pushes that receive a "reject" verdict.
Terraphim Integration (Optional)
When terraphim-cli is available, the judge runner uses knowledge graph-based term normalization to identify rubric dimensions in model reasoning.
Knowledge Graph Setup
# Install judge KG files and configure role
bash automation/judge/setup-judge-kg.sh
# Verify installation
terraphim-cli thesaurus --limit 50
terraphim-cli find "factual correctness and actionability"
KG Files
Located in automation/judge/kg/:
| File | Normalized Term | Purpose |
|---|---|---|
judge-semantic.md |
judge-semantic | Synonyms for semantic quality dimension |
judge-pragmatic.md |
judge-pragmatic | Synonyms for pragmatic quality dimension |
judge-syntactic.md |
judge-syntactic | Synonyms for syntactic quality dimension |
judge-verdicts.md |
judge-verdicts | Verdict vocabulary normalization |
judge-checklist.md |
judge-checklist | Required verdict elements |
Fail-Open Design
If terraphim-cli is not installed, the judge falls back to direct JSON extraction without term normalization. All core functionality works without terraphim -- the KG integration is an enrichment layer.
Resources
references/
- prompt-quick.md -- Quick judge prompt template for fast screening
- prompt-deep.md -- Deep judge prompt template for thorough evaluation
Converted and distributed by TomeVault — claim your Tome and manage your conversions.