Mmdocbench Eval

mmdocbench-eval

MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding — Zhu et al. (2024) (arXiv:2410.21311, 2024)

What this evaluates

Evaluates large vision-language models on fine-grained visual document understanding by testing both answer prediction and multi-granularity visual grounding (region localization) across diverse document types like tables, charts, and infographics.

Datasets

• MMDocBench — total 4338; splits: test (4338)

Metrics

• Exact Match (EM) (primary) — range: percent
  • EM is 1 if every character of the model's text prediction exactly matches the ground truth, otherwise 0.
• F1-score — range: percent
  • Word-level F1 score calculated from the intersection of predicted and ground truth word sets, divided by their total count.
• IOU — range: percent
  • Intersection over Union computed between predicted and ground-truth bounding boxes as overlapping area divided by union area.

Input / output format

Input: A document image paired with a task-specific question and instruction template.

Output: A JSON-formatted string containing an 'answer' field (string) and a 'bbox' field (list of bounding box coordinate strings).

Scoring recipe

pred = extract_json(response)
pred_ans = pred['answer']
pred_bboxes = pred['bbox']
gold_ans = gold['answer']
gold_bboxes = gold['bbox']

em = 1.0 if pred_ans == gold_ans else 0.0
p_words, g_words = set(pred_ans.split()), set(gold_ans.split())
f1 = 2 * len(p_words & g_words) / (len(p_words) + len(g_words)) if (len(p_words) + len(g_words)) > 0 else 0.0

ious = [box_iou(p, g) for p in pred_bboxes for g in gold_bboxes]
iou = max(ious) if ious else 0.0
return em, f1, iou

Common pitfalls

• Models frequently fail to follow JSON instruction templates, requiring strict regex extraction that may discard valid answers.
• Region prediction (IOU) scores are consistently near zero for most models, highlighting a severe grounding deficit rather than answer prediction failure.
• Overall scores use macro averaging across sub-tasks, which can mask performance variations across specific document types like charts or infographics.

Evidence (verbatim from paper)

For each question, we use Exact Match (EM) and $F1$-score to evaluate the predicted answer, and Intersection over Union (IOU) to assess the predicted region(s). The EM is determined by matching every character of the model’s text prediction to the ground truth. If all characters are matched, the EM is 1, and otherwise 0. For $F1$-score, we calculate the word-level F1 based on the number of words in model prediction, ground truth, and their intersection. TheIntersection over Union (IOU) is computed between the predicted region and the ground-truth region, taking into account their overlapping area and union area.

Citation

@misc{zhu2024mmdocbench,
  title={MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding},
  author={Zhu et al. (2024)},
  year={2024},
  note={arXiv:2410.21311}
}

• arXiv: 2410.21311