Vidore V3 Eval

vidore-v3-eval

ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios — Loison et al. (2026) (arXiv:2601.08620, 2026)

What this evaluates

This benchmark evaluates end-to-end Retrieval Augmented Generation (RAG) systems on visually rich, real-world documents across multiple professional domains. It probes a model's ability to retrieve relevant pages, generate accurate answers to complex open-ended and multi-hop queries, and precisely ground those answers with bounding boxes in multimodal content.

Datasets

• ViDoRe V3 — total ?; splits: public (-1), private (-1)

Metrics

• F1 score (Dice coefficient) (primary) — range: [0, 1]
  • Pixel-level overlap metric computed by merging all bounding boxes for a query into a single zone per annotator or model, then comparing zones across annotators. Averaged over pairwise comparisons when multiple annotators label the same sample.
• Intersection over Union (IoU) — range: [0, 1]
  • Ratio of the intersection area to the union area of two bounding box zones, used alongside F1 to measure grounding precision and recall trade-offs.
• Gwet’s AC2 — range: [-1, 1]
  • Statistical measure of inter-annotator agreement that remains stable under prevalence skew, used to validate the quality of query-page relevance labeling.

Input / output format

Input: Text query (question, keyword, or instruction) paired with visually rich document pages containing text, tables, charts, infographics, and images.

Output: Retrieved relevant pages, a generated answer based on those pages, and bounding box coordinates grounding the answer within the document images.

Scoring recipe

def compute_grounding_metrics(pred_boxes, gold_boxes):
    # Merge boxes into zones if multiple per query
    pred_zone = merge_boxes(pred_boxes)
    gold_zone = merge_boxes(gold_boxes)
    iou = intersection_over_union(pred_zone, gold_zone)
    # F1/Dice is derived from IoU or computed via pixel overlap
    f1 = 2 * iou / (iou + 1) if iou > 0 else 0
    return {'IoU': iou, 'F1': f1}

Common pitfalls

• Inter-annotator agreement is moderate (IoU 0.50, F1 0.60), so model performance should be evaluated against human upper bounds rather than assuming perfect gold standards.
• The benchmark uses a split-release strategy with 8 public and 2 private corpora to prevent data contamination; evaluating on private sets requires blind submission protocols.
• Queries often require interpreting non-textual elements (charts, infographics, tables), which current vision-language models frequently fail to ground or reason over accurately.

Evidence (verbatim from paper)

We compute inter-annotator agreement on the subset of query-page pairs labeled by two or three annotators. For each annotator, we merge all their bounding boxes into a single zone. We then compare zones across annotators by measuring pixel-level overlap, reporting Intersection over Union (IoU) and F1 score (Dice coefficient). When three annotators label the same sample, we average over all pairwise comparisons.

Citation

@misc{loison2026vidorev3,
  title={ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios},
  author={Loison et al. (2026)},
  year={2026},
  note={arXiv:2601.08620}
}

• arXiv: 2601.08620