Cogdoc Eval

cogdoc-eval

CogDoc: Towards Unified thinking in Documents — Qixin Xu et al. (2025) (arXiv:2512.12658, 2025)

What this evaluates

Evaluates vision-language models on multi-page document understanding, specifically testing long-context compositional reasoning, fine-grained information extraction from forms, complex layout and chart comprehension, and cross-page navigation for answer localization.

Datasets

• MMLongbench-Doc — total ?; splits: test (-1)
• DUDE — total ?; splits: test (-1)
• SlideVQA — total ?; splits: test (-1)
• MP-DocVQA — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Exact match ratio: 1 if the predicted answer or retrieved page set exactly matches the ground truth, 0 otherwise.
• ANLS — range: [0, 1]
  • Average Normalized Levenshtein Similarity between predicted and ground-truth text spans, normalized by the length of the longer string.
• F1 — range: [0, 1]
  • Token-level F1 score computed between predicted and ground-truth answers.

Input / output format

Input: Multi-page document images paired with natural language queries requiring reasoning, information extraction, or page navigation.

Output: Sequential two-stage output: first, a set of retrieved page identifiers; second, a final textual answer or extracted value.

Scoring recipe

def compute_metrics(predictions, golds):
    acc = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    anls = sum(normalized_levenshtein(p, g) for p, g in zip(predictions, golds)) / len(golds)
    f1 = compute_token_f1(predictions, golds)
    return {'Accuracy': acc, 'ANLS': anls, 'F1': f1}

Common pitfalls

• Assuming the SFT+RL training strategy universally outperforms Direct RL; the paper shows SFT+RL degrades performance on long-context benchmarks due to internal policy conflict.
• Treating MP-DocVQA results as in-distribution performance; the model was explicitly not trained on its trainset, so scores measure zero-shot generalization to unseen document types.
• Confusing Stage 1 retrieval metrics (page ID accuracy/recall) with Stage 2 answer metrics (ANLS/Accuracy/F1) when comparing ablation variants.

Evidence (verbatim from paper)

Acc. (Accuracy) indicates if the retrieved page identifiers set is an exact match to the ground truth, while Rec. (Recall) indicates if the retrieved set fully includes all relevant page numbers.

Citation

@misc{xu2025cogdoc,
  title={CogDoc: Towards Unified thinking in Documents},
  author={Qixin Xu et al. (2025)},
  year={2025},
  note={arXiv:2512.12658}
}

• arXiv: 2512.12658