Validate Structured Model Output
Treat model output as untrusted input. Preserve raw bytes and validate a normalized copy through ordered gates.
Validation order
- Transport: A native structured response or the required JSON payload exists.
- Parse: Decode exactly once. In acceptance benchmarks, do not extract JSON from prose or retry malformed output.
- Schema: Enforce required fields, types, controlled values, and cardinality.
- Identifiers: Reject unknown, duplicate, overlapping, invented, or off-request IDs.
- Semantics: Reject equivalent choices where distinctness is required and verify classifications against supplied facts.
- Execution: Replay or otherwise validate the selected operation with the deterministic kernel.
- Safety: Reject continuation through terminal, unresolved, random, hidden, or otherwise unsupported effects.
Keep failure categories separate. A parse failure is not an abstention, and a legal identifier is not proof of a safe or useful result.
Validate JSONL ID contracts
Use the bundled validator when each row contains supplied, ranked, and optional alternative IDs:
python scripts/validate_jsonl_outputs.py results.jsonl \
--supplied-field request.planIDs \
--ranked-field response.rankedPlanIDs \
--alternative-field response.alternativePlanIDs \
--max-ranked 3 --output validation-report.json
It checks duplicate IDs within arrays, overlap between ranked and alternatives, unknown IDs, rank length, and alternatives without a ranked selection. Add project validators after this generic layer.
Benchmark discipline
For first-attempt acceptance, generate once and validate without repair, remediation, or deterministic fallback. Test production remediation separately. Persist raw output, normalized output, every gate result, timing, prompt/config revision, and validator revision so failures remain auditable.
1---2name: validate-structured-model-output3description: Validates JSON or tool-call model outputs without repair, separating parse, schema, supplied-identifier, semantic, replay, and safety failures. Use for first-attempt benchmarks, teacher-label pipelines, and bounded model tool selection.4---56# Validate Structured Model Output78Treat model output as untrusted input. Preserve raw bytes and validate a normalized copy through ordered gates.910## Validation order11121. **Transport:** A native structured response or the required JSON payload exists.132. **Parse:** Decode exactly once. In acceptance benchmarks, do not extract JSON from prose or retry malformed output.143. **Schema:** Enforce required fields, types, controlled values, and cardinality.154. **Identifiers:** Reject unknown, duplicate, overlapping, invented, or off-request IDs.165. **Semantics:** Reject equivalent choices where distinctness is required and verify classifications against supplied facts.176. **Execution:** Replay or otherwise validate the selected operation with the deterministic kernel.187. **Safety:** Reject continuation through terminal, unresolved, random, hidden, or otherwise unsupported effects.1920Keep failure categories separate. A parse failure is not an abstention, and a legal identifier is not proof of a safe or useful result.2122## Validate JSONL ID contracts2324Use the bundled validator when each row contains supplied, ranked, and optional alternative IDs:2526```sh27python scripts/validate_jsonl_outputs.py results.jsonl \28 --supplied-field request.planIDs \29 --ranked-field response.rankedPlanIDs \30 --alternative-field response.alternativePlanIDs \31 --max-ranked 3 --output validation-report.json32```3334It checks duplicate IDs within arrays, overlap between ranked and alternatives, unknown IDs, rank length, and alternatives without a ranked selection. Add project validators after this generic layer.3536## Benchmark discipline3738For first-attempt acceptance, generate once and validate without repair, remediation, or deterministic fallback. Test production remediation separately. Persist raw output, normalized output, every gate result, timing, prompt/config revision, and validator revision so failures remain auditable.
bastos