Vidore Eval

vidore-eval

ColPali: Efficient Document Retrieval with Vision Language Models — Faysse et al. (2024) (arXiv:2407.01449, 2024)

What this evaluates

Evaluates page-level document retrieval on visually rich documents across diverse domains and languages. It probes the model's ability to leverage visual cues, layout, and text within document images without relying on traditional OCR or layout parsing pipelines.

Datasets

• ViDoRe — total ?; splits: test (-1)

Metrics

• nDCG@5 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 5. It measures the quality of the ranked list of retrieved document pages relative to ground truth relevant pages, discounting relevance by position using log2(i+2).

Input / output format

Input: A natural language query and a document page image (or set of images).

Output: A ranked list of document pages or relevance scores computed via late interaction between query tokens and image patch embeddings.

Scoring recipe

def ndcg_at_5(retrieved_docs, relevant_docs, k=5):
    dcg = 0.0
    for i, doc in enumerate(retrieved_docs[:k]):
        rel = 1.0 if doc in relevant_docs else 0.0
        dcg += rel / math.log2(i + 2)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(relevant_docs), k)))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• The benchmark evaluates page-level retrieval rather than chunk-level, so models relying on text chunking or OCR pipelines require adaptation.
• nDCG@5 is used instead of the more common nDCG@10 or MRR, so direct comparison with other retrieval papers requires careful metric alignment.
• Late interaction matching computes scores per patch/token, which significantly increases memory and compute compared to bi-encoders, affecting latency benchmarks.

Evidence (verbatim from paper)

Results are presented using nDCG@5 metrics, and illustrate the impact of different components. Text-only metrics are not computed for benchmarks with only visual elements.

Citation

@misc{faysse2024colpali,
  title={ColPali: Efficient Document Retrieval with Vision Language Models},
  author={Faysse et al. (2024)},
  year={2024},
  note={arXiv:2407.01449}
}

• arXiv: 2407.01449