Gdibench Eval

gdibench-eval

GDI-Bench: A Benchmark for General Document Intelligence with Vision and Reasoning Decoupling — Siqi Li et al. (2025) (arXiv:2505.00063, 2025)

What this evaluates

Evaluates document intelligence by decoupling visual and reasoning complexity into graded difficulty levels (V0–V2, R0–R2). It probes a model’s ability to extract, reason over, and generalize across diverse document types while mitigating catastrophic forgetting during fine-tuning.

Datasets

• GDI-Bench — total ?; splits: test (-1)

Metrics

• Accuracy / normalized edit distance (primary) — range: [0, 1]
  • For GDI-Bench, accuracy is computed as the percentage of correctly answered questions or extracted fields across vision (V0-V2) and reasoning (R0-R2) complexity levels. For cross-domain tasks (T1-T4), normalized edit distance measures the similarity between predicted and gold text sequences.

Input / output format

Input: Document images paired with task-specific prompts or questions, processed through a standardized preprocessing pipeline.

Output: Text responses containing extracted information, answers to questions, or formatted document fields.

Scoring recipe

def compute_metric(predictions, golds, metric_type='accuracy'):
    if metric_type == 'accuracy':
        correct = sum(1 for p, g in zip(predictions, golds) if normalize(p) == normalize(g))
        return correct / len(golds)
    elif metric_type == 'normalized_edit_distance':
        distances = [edit_distance(p, g) / max(len(p), len(g)) for p, g in zip(predictions, golds)]
        return sum(distances) / len(distances)

Common pitfalls

• Difficulty levels (V0-V2, R0-R2) are decoupled, so models may excel in vision but fail in reasoning or vice versa.
• OmniDocBench uses a lower-is-better metric (↓), unlike other benchmarks in the suite.
• Cross-domain/cross-task evaluations (T1-T4) use normalized edit distance, not standard accuracy.

Evidence (verbatim from paper)

We compare the performance of Full-Parameter Fine-Tuning, LoRA Fine-Tuning, and the LW-AFT method under both settings, as shown in Table [4], which displays the normalized edit distance for each task.

Citation

@misc{li2025gdibench,
  title={GDI-Bench: A Benchmark for General Document Intelligence with Vision and Reasoning Decoupling},
  author={Siqi Li et al. (2025)},
  year={2025},
  note={arXiv:2505.00063}
}

• arXiv: 2505.00063