Llmstructbench Eval

llmstructbench-eval

LLMStructBench: Benchmarking Large Language Model Structured Data Extraction — Tenckhoff et al. (2026) (arXiv:2602.14743, 2026)

What this evaluates

Evaluates large language models' ability to extract structured data from natural-language emails into valid JSON objects that adhere to a provided schema. It measures both syntactic validity (structural correctness) and semantic accuracy (correct value extraction) across varying levels of JSON nesting complexity.

Datasets

• LLMStructBench — total 995; splits: test (995)

Metrics

• DOC (primary) — range: [0, 1]
  • DOC measures document-level structural compliance, evaluating whether the generated JSON strictly adheres to the provided schema and is syntactically valid.
• MK — range: [0, 1]
  • MK evaluates key-level matching accuracy, measuring the proportion of correctly extracted keys relative to the ground truth.
• MV — range: [0, 1]
  • MV measures value-level matching accuracy, assessing the correctness of extracted values against the ground truth.
• WV — range: [0, 1]
  • WV computes a weighted value match score, accounting for the importance or frequency of specific fields in the extraction task.

Input / output format

Input: Natural-language message, a representative example input-output pair, and a JSON schema defining required keys, data types, and nesting depth.

Output: A single JSON object adhering strictly to the provided schema.

Scoring recipe

def evaluate(gen_output, gt_json, schema):
    try:
        parsed = json.loads(gen_output)
        is_valid_schema = validate_schema(parsed, schema)
    except json.JSONDecodeError:
        return {'DOC': 0.0, 'MK': 0.0, 'MV': 0.0, 'WV': 0.0}
    mk = compute_key_match(parsed, gt_json)
    mv = compute_value_match(parsed, gt_json)
    wv = compute_weighted_value_match(parsed, gt_json)
    return {'DOC': float(is_valid_schema), 'MK': float(mk), 'MV': float(mv), 'WV': float(wv)}

Common pitfalls

• Confusing syntactic validity (schema compliance/valid JSON) with semantic accuracy (correct extraction of values from text).
• Assuming model size is the main performance driver; the benchmark shows prompting strategy (e.g., schema-driven or two-step) has a greater impact on parsing reliability.
• Treating the dataset as purely real-world; it is synthetically generated from structured JSON via GPT-4o and manually verified, which may not capture all real-world email noise.

Evidence (verbatim from paper)

The benchmark introduces novel metrics (e.g., DOC, MK, MV, WV) that separate syntactic validity from semantic accuracy, enabling fine-grained analysis of model and prompting performance in end-to-end information extraction workflows.

Citation

@misc{tenckhoff2026llmstructbench,
  title={LLMStructBench: Benchmarking Large Language Model Structured Data Extraction},
  author={Tenckhoff et al. (2026)},
  year={2026},
  note={arXiv:2602.14743}
}

• arXiv: 2602.14743