Jsonschemabench Coverage Eval

jsonschemabench-coverage-eval

JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models — Geng et al. (2025) (arXiv:2501.10868, 2025)

What this evaluates

Evaluates whether language models can generate JSON objects that strictly comply with a given JSON Schema under constrained decoding. It measures the empirical coverage of schema features supported by the model and decoding framework.

Datasets

• JSONSchemaBench — total 10000; splits: test (10000); repo https://github.com/guidance-ai/jsonschemabench

Metrics

• Top 1 Empirical Coverage (primary) — range: percent
  • The proportion of schemas for which the single top-1 generated JSON object passes validation against the provided JSON Schema (Draft-2020-12 with format checks enabled).

Input / output format

Input: A JSON Schema (Draft-2020-12) embedded in a prompt template.

Output: A single JSON object generated via greedy decoding.

Scoring recipe

compliant = 0
for schema, gen_json in zip(schemas, outputs):
    try:
        jsonschema.validate(gen_json, schema, format_checker=jsonschema.FormatChecker())
        compliant += 1
    except jsonschema.ValidationError:
        pass
coverage = compliant / len(schemas)

Common pitfalls

• Only generating a single top-1 output may underestimate true model coverage, as sampling multiple outputs could reveal compliance on harder schemas.
• The jsonschema library does not guarantee validation of all schema constraints, even with format checks enabled, potentially leading to a slight overestimation of compliance.

Evidence (verbatim from paper)

We use greedy decoding with no top P or top K sampling for all the experiments. We only get one output from the model, which we will use to validate the schema compliance. It’s totally plausible to sample more outputs and validate them all, and it might detect more schema violations. The fact that we only sample the top 1 output may quantify our empirical coverage as Top 1 Empirical Coverage. We use the jsonschema library with the Draft-2020-12 version of the JSON Schema standard to validate the generated JSON object. We turn on the ‘format’ checks, which are not enabled by default in Python. Strictly speaking, the jsonschema library doesn’t guarantee the validation of all the schema constraints, even with the ‘format’ checks enabled. It is possible, though very rare, for a schema-noncompliant output to be validated as compliant by the jsonschema library, leading to a slight overestimation of empirical coverage. However, such occurrences are corner cases and happen infrequently.

Citation

@misc{geng2025jsonschemabench,
  title={JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models},
  author={Geng et al. (2025)},
  year={2025},
  note={arXiv:2501.10868}
}

• arXiv: 2501.10868