Docgenome Eval

docgenome-eval

DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Models — Xia et al. (2024) (arXiv:2406.11633, 2024)

What this evaluates

This benchmark evaluates multi-modal large language models on their ability to parse and understand complex scientific documents. It probes capabilities across document classification, visual grounding of text elements, open-ended single- and multi-page question answering, layout detection, and modality-to-LaTeX transformation.

Datasets

• DocGenome — total ?; splits: train (-1), test (-1); repo https://github.com/UniModal4Reasoning/DocGenome

Metrics

• Accuracy — range: [0, 1]
  • Proportion of correctly classified documents out of the total test set.
• Edit Distance — range: [0, 1]
  • Normalized Levenshtein distance between predicted and ground-truth text or LaTeX strings, where lower values indicate better alignment.
• GPT-acc (primary) — range: [0, 1]
  • Accuracy score computed by an LLM judge (GPT-4) evaluating the semantic correctness of open-ended QA answers against gold references.
• mAP@0.5:0.95 — range: [0, 1]
  • Mean Average Precision averaged over Intersection over Union (IoU) thresholds ranging from 0.5 to 0.95 for layout component detection.
• Jaccard Similarity — range: [0, 1]
  • Intersection over union of token sets between predicted and ground-truth LaTeX or code outputs.
• Cosine Similarity — range: [0, 1]
  • Cosine similarity between sentence embeddings of predicted and ground-truth outputs.
• BLEU — range: [0, 1]
  • n-gram precision score with a brevity penalty applied to generated LaTeX or code.

Input / output format

Input: Document images (single-page for most tasks, multi-page for multi-page QA) combined with instruction prompts. For layout detection and transformation subtasks, images are often cropped to isolate specific modalities (e.g., equations, tables).

Output: Task-specific predictions: class labels, bounding box coordinates, natural language answers, or LaTeX source code.

Scoring recipe

def compute_metrics(predictions, gold, task):
    if task == 'classification':
        return sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    elif task in ['grounding', 'transformation']:
        return normalized_edit_distance(predictions, gold)
    elif task == 'qa':
        return llm_judge_accuracy(predictions, gold)  # GPT-acc
    elif task == 'layout':
        return mean_ap(predictions, gold, iou_thresh=[0.5, 0.95])
    elif task == 'transformation':
        return {
            'edit_distance': normalized_edit_distance(predictions, gold),
            'jaccard': jaccard_similarity(predictions, gold),
            'cosine': cosine_similarity(predictions, gold),
            'bleu': bleu_score(predictions, gold)
        }

Common pitfalls

• GPT-acc relies on an external LLM judge, making scores sensitive to the judge model's version, prompt template, and temperature settings.
• Edit Distance for LaTeX/OCR penalizes semantically equivalent but syntactically different formatting, potentially underestimating model capability.
• mAP@0.5:0.95 for layout detection requires precise bounding box annotations; minor annotation shifts or overlapping components can significantly degrade scores.

Evidence (verbatim from paper)

We evaluate the performance of several state-of-the-art multi-modal large language models on the proposed DocGenome-test, covering document classification, visual grounding, and both single-page and multi-page QA tasks. As shown in Table [3], among the tested models, GPT-4V*[[33]]* achieves the highest classification accuracy with 98.0% Top-1 Acc, while QWen-VL*[[5]]* and InternVL 1.5*[[8]]* also show competitive results with 82.4% and 75.9% accuracy, respectively. For the visual grounding task, GPT4V showcases the best performance in the Title OCR Grounding task with the lowest Edit Distance of 0.0104, while InternVL 1.5 outperforms other models in the Abstract OCR Grounding task with the lowest Edit Distance of 0.3601. In the single-page QA task, GPT-4V attains the highest GPT-acc score of 61.0%, indicating its superior ability to handle document-based QA tasks.

Citation

@misc{xia2024docgenome,
  title={DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Models},
  author={Xia et al. (2024)},
  year={2024},
  note={arXiv:2406.11633}
}

• arXiv: 2406.11633