Deepjsoneval Eval

deepjsoneval-eval

DeepJSONEval: Benchmarking Complex Nested JSON Data Mining for Large Language Models — Zhou et al. (2025) (arXiv:2509.25922, 2025)

What this evaluates

Evaluates LLMs' ability to extract and structure information from unstructured text into deep, multi-layer nested JSON formats. It probes format fidelity, field correctness, and structural completeness across varying nesting depths and domains.

Datasets

• DeepJSONEval — total 2100; splits: test (2100); repo https://github.com/GTS-AI-Infra-Lab-SotaS/DeepJSONEval

Metrics

• detailed score (primary) — range: [0, 1]
  • Measures comprehensive extraction capabilities by combining format fidelity, field correctness, and structural completeness. Exact weighting and calculation are detailed in the Appendix.
• strict score — range: [0, 1]
  • Binary or normalized score requiring complete accuracy; used for agentic systems where any extraction error is unacceptable.
• format score — range: [0, 1]
  • Evaluates structural fidelity and adherence to the nested JSON schema without necessarily checking semantic field correctness.

Input / output format

Input: Unstructured text paired with a multi-layer nested JSON schema (3–7 levels deep, ~17.5 properties) specifying target data types (strings, numbers, booleans, enumerations, lists).

Output: A syntactically valid nested JSON object that conforms to the provided schema and contains the extracted information.

Scoring recipe

# Pseudo-code based on paper description (exact formulas in Appendix)
def compute_metrics(pred_json, gold_json, schema):
    format_score = check_syntax_and_structure(pred_json, schema)
    field_correctness = compare_values(pred_json, gold_json)
    structural_completeness = check_missing_keys(pred_json, schema)
    detailed_score = aggregate(format_score, field_correctness, structural_completeness)
    strict_score = 1.0 if (format_score == 1.0 and field_correctness == 1.0) else 0.0
    return detailed_score, strict_score, format_score

Common pitfalls

• Assuming response length correlates with performance (paper shows R² < 0.05).
• Equating schema adherence with successful extraction (benchmark explicitly measures beyond format to true information extraction).
• Ignoring difficulty stratification (strict scores drop 17-37% on hard tasks vs medium, revealing capability gaps).

Evidence (verbatim from paper)

The detailed score measures the comprehensive extraction capabilities of models, with higher scores indicating stronger extraction performance. However, in specific agentic systems where complete accuracy is required, the strict score provides a more accurate reflection of model capabilities.

Citation

@misc{zhou2025deepjsoneval,
  title={DeepJSONEval: Benchmarking Complex Nested JSON Data Mining for Large Language Models},
  author={Zhou et al. (2025)},
  year={2025},
  note={arXiv:2509.25922}
}

• arXiv: 2509.25922