Vidoseek Eval

vidoseek-eval

ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agents — Qiuchen Wang et al. (2025) (arXiv:2502.18017, 2025)

What this evaluates

Evaluates a multi-agent RAG framework's ability to retrieve relevant pages from visually rich documents and generate accurate answers through iterative reasoning. It probes hybrid visual-textual retrieval and dynamic token allocation for document comprehension.

Datasets

• ViDoSeek — total 6000; splits: test (-1); repo https://github.com/Alibaba-NLP/ViDoRAG

Metrics

• accuracy (primary) — range: [0, 1]
  • Model-based evaluation using GPT-4o to score generated answers against references on a 1-5 scale. Accuracy is the proportion of answers scoring 4 or higher.
• Recall@K — range: percent
  • Standard recall at K=1, 3, 5 measuring the fraction of queries where the golden page appears in the top-K retrieved results. Also reports MRR@5.

Input / output format

Input: Visual document images and natural language queries.

Output: Final generated answer (for generation task) or retrieved document/page IDs (for retrieval task).

Scoring recipe

# Generation Accuracy
scores = gpt4o_score(reference_answer, generated_answer)  # 1-5 scale
accuracy = sum(1 for s in scores if s >= 4) / len(scores)

# Retrieval Recall@K
recall_at_k = len(set(retrieved_top_k) & set(golden_pages)) / len(golden_pages)

Common pitfalls

• GPT-4o scoring uses a threshold of 4/5 for correctness rather than exact string matching.
• Retrieval baselines are normalized to the average dynamic recall length of the proposed method to ensure fair comparison, rather than using a fixed top-K across all methods.

Evidence (verbatim from paper)

For our end-to-end evaluation, we employed a model-based assessment using GPT-4o, which involved assigning scores from 1 to 5 by comparing the reference answer with the final answer. Answers receiving scores of 4 or above were considered correct, and we subsequently calculate accuracy as the evaluation metric. For retrieval evaluation, we use recall as the metric.

Citation

@misc{wang2025vidorag,
  title={ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agents},
  author={Qiuchen Wang et al. (2025)},
  year={2025},
  note={arXiv:2502.18017}
}

• arXiv: 2502.18017