Extractbench Eval

extractbench-eval

ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction — Ferguson et al. (2026) (arXiv:2602.12247, 2026)

What this evaluates

Evaluates LLMs on complex, schema-driven PDF-to-JSON structured extraction, testing their ability to handle nested objects, arrays, heterogeneous field types, and strict correctness criteria across enterprise-scale documents.

Datasets

• ExtractBench — total 35; splits: benchmark (35); repo https://github.com/ContextualAI/extract-bench

Metrics

• Pass Rate (primary) — range: percent
  • Field-level pass rate where invalid extractions contribute 0. Computed as the fraction of fields that pass their schema-defined metric.
• Valid JSON — range: percent
  • Fraction of model outputs that are parseable and conform to the provided JSON Schema.
• Acc (Valid) — range: percent
  • Pass rate conditioned only on extractions that produced valid JSON.
• string_semantic — range: [0, 1]
  • LLM-based equivalence metric for free text, allowing domain-specific instructions for acceptable variations.
• number_tolerance — range: [0, 1]
  • Numeric comparison within a specified margin (e.g., 0.1%).
• array_llm — range: percent
  • Semantic alignment of object arrays using an LLM judge to map matched pairs, missed gold items, and spurious predictions, then computing precision, recall, and F1.

Input / output format

Input: PDF documents paired with a JSON Schema that defines the target extraction structure and per-field evaluation configurations.

Output: A JSON object containing extracted fields matching the schema structure. Must be parseable and schema-conforming to be considered valid.

Scoring recipe

def evaluate(gold, pred, schema):
    if not is_valid_json(pred, schema):
        return {"valid_json": False, "pass_rate": 0.0}
    scores = []
    for field in schema.fields:
        g_val = gold.get(field.name)
        p_val = pred.get(field.name)
        if g_val is None and p_val is None:
            scores.append(True)
        elif g_val is None and p_val is not None:
            scores.append(False) # hallucination
        elif g_val is not None and p_val is None:
            scores.append(False) # omission
        else:
            scores.append(field.metric(g_val, p_val, field.config))
    return {"valid_json": True, "pass_rate": sum(scores)/len(scores)}

Common pitfalls

• Applying a single global metric (e.g., exact match) across all fields ignores heterogeneous correctness criteria like numeric tolerance or semantic equivalence.
• Ignoring the three-way missing value distinction (present, null, MISSING) conflates omissions with hallucinations and structural errors.
• Using position-based matching for arrays fails when models reorder, omit, or duplicate items, leading to artificially low scores.

Evidence (verbatim from paper)

Pass Rate: field-level pass rate (invalid extractions contribute 0). Acc (Valid): pass rate conditioned on valid extraction only.

Citation

@misc{ferguson2026extractbench,
  title={ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction},
  author={Ferguson et al. (2026)},
  year={2026},
  note={arXiv:2602.12247}
}

• arXiv: 2602.12247