Unidoc Bench Eval

unidoc-bench-eval

UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG — Peng et al. (2025) (arXiv:2510.03663, 2025)

What this evaluates

Evaluates retrieval and end-to-end generation performance of multimodal RAG systems on real-world PDF documents. It probes the ability of text-only, image-only, and multimodal (text-image fusion/joint) retrieval paradigms to locate relevant evidence and generate faithful, complete answers to cross-modality questions.

Datasets

• UNIDOC-BENCH — total 1600; splits: test (1600); repo https://github.com/SalesforceAIResearch/UniDOC-Bench

Metrics

• Precision@10 (primary) — range: [0, 1]
  • Fraction of the top-10 retrieved candidates that match the ground-truth context in both page number and file. A true positive requires exact page and file match.
• Recall@10 (primary) — range: [0, 1]
  • Fraction of ground-truth contexts successfully retrieved within the top-10 candidates. Partial page overlaps are counted as true positives.
• Faithfulness (primary) — range: [0, 1]
  • LLM-based metric measuring whether the facts extracted from the system's response are grounded in the ground-truth chunks. Calculated as the proportion of response facts verified against ground-truth evidence.
• Completeness (primary) — range: [0, 1]
  • LLM-based metric measuring whether the facts required to answer the question (extracted from the ground-truth answer) appear in the system's response. Calculated as the proportion of ground-truth facts present in the response.

Input / output format

Input: Per instance: a question, a set of PDF documents (provided as text chunks and/or page-level JPEG images), and ground-truth answer/context metadata.

Output: Per instance: top-10 retrieved candidates (text chunks or images) for retrieval evaluation; a final generated text response for end-to-end evaluation.

Scoring recipe

def score_retrieval(preds, gold):
    tp = sum(1 for p in preds[:10] if p.page == gold.page and p.file == gold.file)
    return tp / 10, tp / len(gold)

def score_e2e(response, gold_answer, gold_chunks, llm_judge):
    resp_facts = llm_judge.extract_facts(response)
    gold_facts = llm_judge.extract_facts(gold_answer)
    faithfulness = len([f for f in resp_facts if f in gold_chunks]) / len(resp_facts)
    completeness = len(set(resp_facts) & set(gold_facts)) / len(gold_facts)
    return faithfulness, completeness

Common pitfalls

• Partial page overlaps (e.g., retrieved pages 1–3 vs. ground-truth pages 3–5) are counted as true positives, which slightly inflates Recall@10 scores.
• Absolute metric values should not be overinterpreted; the benchmark is designed to highlight relative performance differences across retrieval paradigms rather than provide absolute capability baselines.
• Image-only retrieval often achieves higher recall at the retrieval stage but fails to translate to better end-to-end completeness due to LLM processing limitations on page-level images.

Evidence (verbatim from paper)

Specifically, we first ask the LLM to extract the facts required to answer each question and then verify whether these facts are grounded in the ground-truth chunks; this is measured as faithfulness. Next, we ask the LLM to extract the facts required to answer the question from the ground-truth answer and then check whether each fact appears in the system’s response; this is measured as completeness. Higher faithfulness and completeness scores are better.

Citation

@misc{peng2025unidocbench,
  title={UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG},
  author={Peng et al. (2025)},
  year={2025},
  note={arXiv:2510.03663}
}

• arXiv: 2510.03663