qhjqhj00 structured-output-benchmark-eval
The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models — Singh et al. (2026) (arXiv:2604.25359, 2026)
What this evaluates
Evaluates large language models' ability to extract structured information from multi-modal sources (text, images, audio) into valid JSON formats, isolating schema compliance from value accuracy.
Datasets
- Multi-Source Structured Output Benchmark — total ?; splits: test (-1)
Metrics
correct_value_extraction(primary) — range: [0, 1]- Measures the correctness of extracted values against ground truth across text, image, and audio sources, independent of schema validity.
Input / output format
Input: Context (text/image/audio), question, and a JSON schema defining the expected output structure.
Output: A conforming JSON response containing the extracted structured data.
Scoring recipe
def score(predictions, gold, schema):
schema_valid = validate_json_schema(predictions, schema)
value_correct = exact_match(predictions, gold)
return {
'schema_compliance': schema_valid,
'correct_value_extraction': value_correct,
'overall_fidelity': schema_valid and value_correct
}
Common pitfalls
- Hallucinations in structured fields are harder to detect because syntactically correct JSON can still contain incorrect values.
- Model size does not correlate with extraction performance, contradicting typical scaling expectations.
- Evaluating in reasoning mode conflates extraction capability with compute budget and problem-solving ability.
Evidence (verbatim from paper)
It reveals a stark gap between schema validity and correct value extraction—83.0% on text, 67.2% on images, 23.7% on audio—showing that model size does not correlate with performance and that hallucinations in structured fields are harder to detect due to syntactic correctness.
Citation
@misc{singh2026sob,
title={The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models},
author={Singh et al. (2026)},
year={2026},
note={arXiv:2604.25359}
}
- arXiv: 2604.25359