Scimdr Eval

scimdr-eval

SciMDR: Benchmarking and Advancing Scientific Multimodal Document Reasoning — Chen et al. (2026) (arXiv:2603.12249, 2026)

What this evaluates

Evaluates a model's ability to perform complex, claim-centric reasoning over full scientific documents containing multimodal elements (charts, tables, figures). It probes the model's capacity to localize evidence and answer questions accurately despite long-context noise and distractors.

Datasets

• SciMDR-Eval — total 300000; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions on the test set, computed as exact-match or semantic match depending on the baseline implementation.

Input / output format

Input: Full scientific research papers (text and embedded figures/tables) paired with claim-centric questions requiring multi-step reasoning.

Output: Natural language text answers to the posed questions.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        if normalize_text(pred) == normalize_text(gold):
            correct += 1
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Long-context noise causes severe attention dilution; performance drops significantly when full papers are provided vs. oracle context.
• Simple QA pairs without explicit reasoning chains fail to teach complex scientific logic, causing massive accuracy drops.
• Synthetic data generation must balance faithfulness to source claims with realism of full-document context to avoid training degradation.

Evidence (verbatim from paper)

Table 7: Challenge of Attention Dilution. Effect of context noise on SciMDR-Eval accuracy. Performance degrades as the amount of irrelevant context increases.

Citation

@misc{chen2026scimdr,
  title={SciMDR: Benchmarking and Advancing Scientific Multimodal Document Reasoning},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2603.12249}
}

• arXiv: 2603.12249