Varex Eval

varex-eval

VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents — Barzelay et al. (2026) (arXiv:2603.15118, 2026)

What this evaluates

Evaluates multi-modal structured data extraction from documents, testing a model's ability to parse visual or textual layouts, adhere to a provided JSON schema, and generate compliant structured outputs. It specifically probes schema compliance, layout understanding, and cross-modal robustness across plain text, spatial text, and image inputs.

Datasets

• VAREX — total 1777; splits: test (1777); repo https://github.com/udibarzi/varex-bench

Metrics

• exact match (EM) (primary) — range: [0, 1]
  • A field scores 1 if the normalized prediction exactly matches the normalized ground truth, and 0 otherwise. For array fields, order-invariant matching is applied via the Hungarian algorithm to assign predicted elements to ground-truth elements by maximum overlap.
• ANLS (Average Normalized Levenshtein Similarity) — range: [0, 1]
  • Assigns partial credit for near-matches by computing the normalized Levenshtein similarity between the normalized prediction and ground truth, distinguishing complete misses from minor formatting differences.

Input / output format

Input: A minimal zero-shot prompt instructing the model to extract structured data matching a provided schema and return valid JSON. Inputs are provided in one of four modalities: Plain Text (P), Spatial Text (S), Image (V at 200 or 50 DPI), or Spatial Text + Image (S+V).

Output: Valid JSON object matching the provided schema, with null values for missing fields. Models are instructed to use response_format: {"type": "json_object"}.

Scoring recipe

def compute_metrics(predictions, ground_truth):
    em_scores = []
    anls_scores = []
    for pred, gt in zip(predictions, ground_truth):
        pred_norm = normalize_text(pred)
        gt_norm = normalize_text(gt)
        if pred_norm == gt_norm:
            em_scores.append(1.0)
            anls_scores.append(1.0)
        else:
            em_scores.append(0.0)
            anls_scores.append(normalized_levenshtein_similarity(pred_norm, gt_norm))
    return {
        'EM': sum(em_scores) / len(em_scores),
        'ANLS': sum(anls_scores) / len(anls_scores)
    }

Common pitfalls

• Schema echo: Models below 4B parameters often fail on output compliance/formatting rather than actual extraction, causing 45–65 percentage point score drops.
• Array ordering: Models may traverse table rows in different orders; failing to use order-invariant matching (Hungarian algorithm) unfairly penalizes correct extractions.
• Modality bias: Performance varies significantly across P, S, V, and S+V inputs; comparing scores across modalities without accounting for layout preservation or DPI differences can misrepresent model capability.

Evidence (verbatim from paper)

We report exact match (EM) as the primary metric: a field scores 1 if the normalized prediction exactly matches the normalized ground truth, 0 otherwise. We additionally report ANLS (Average Normalized Levenshtein Similarity), which assigns partial credit for near-matches, to distinguish complete misses from minor formatting differences. For array fields, we apply order-invariant matching via the Hungarian algorithm: predicted array elements are optimally assigned to ground-truth elements by maximum field overlap rather than positional index, ensuring models are not penalized for reading table rows in a different traversal order.

Citation

@misc{barzelay2026varex,
  title={VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents},
  author={Barzelay et al. (2026)},
  year={2026},
  note={arXiv:2603.15118}
}

• arXiv: 2603.15118