Irpapers Eval

irpapers-eval

IRPAPERS: A Visual Document Benchmark for Scientific Retrieval and Question Answering — Shorten et al. (2026) (arXiv:2602.17687, 2026)

What this evaluates

Evaluates the ability of multimodal and text-only models to retrieve relevant scientific paper pages and answer questions based on those pages. It probes retrieval depth, modality complementarity, and the impact of context quantity on RAG performance.

Datasets

• IRPAPERS — total 3230; splits: test (-1)

Metrics

• Recall@1 (primary) — range: [0, 1]
  • Fraction of queries where the single top-ranked retrieved page is the ground-truth relevant page.
• Recall@5 — range: [0, 1]
  • Fraction of queries where the ground-truth relevant page appears in the top 5 retrieved results.
• Recall@20 — range: [0, 1]
  • Fraction of queries where the ground-truth relevant page appears in the top 20 retrieved results.
• Alignment Score — range: [0, 1]
  • Normalized metric measuring how closely the generated answer aligns with the ground truth, reported as a decimal between 0 and 1.

Input / output format

Input: For retrieval: a natural language query and a corpus of scientific paper pages (each containing an image and OCR text). For QA: a question paired with k retrieved pages provided either as raw images or OCR-transcribed text.

Output: For retrieval: a ranked list of page IDs. For QA: a natural language answer string.

Scoring recipe

def calc_recall_at_k(predictions, gold_ids, k):
    hits = sum(1 for pred in predictions if any(g in pred[:k] for g in gold_ids))
    return hits / len(predictions)

def calc_alignment_score(generated, gold):
    # Computed via LLM-judge or semantic similarity per paper protocol
    return sum(align(g, gt) for g, gt in zip(generated, gold)) / len(generated)

Common pitfalls

• Assuming single-modality retrieval is sufficient; the paper demonstrates complementary failure modes that require multimodal fusion to break performance ceilings.
• Using k=1 for question answering when k=5 significantly outperforms oracle k=1, indicating that contextual support from related pages is critical for accurate answer synthesis.
• Ignoring the performance-efficiency tradeoff of compression methods like MUVERA, where higher ef values recover fine-grained matching signals at the cost of computation.

Evidence (verbatim from paper)

Multimodal hybrid search with RSF achieves 49% Recall@1, a 3-point absolute improvement over hybrid text search alone. At Recall@5, multimodal hybrid search achieves a 3-point absolute improvement over the next highest performing open-source method, and a 2-point absolute improvement at Recall@20.

Citation

@misc{shorten2026irpapers,
  title={IRPAPERS: A Visual Document Benchmark for Scientific Retrieval and Question Answering},
  author={Shorten et al. (2026)},
  year={2026},
  note={arXiv:2602.17687}
}

• arXiv: 2602.17687