Vdr Compression Eval

vdr-compression-eval

Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Framework — Yan et al. (2026) (arXiv:2602.19549, 2026)

What this evaluates

Evaluates multi-vector visual document retrieval (VDR) models under varying compression ratios. It measures how well pruning and merging strategies maintain retrieval accuracy while reducing storage and computational overhead.

Datasets

• ViDoRe-V1 — total ?; splits: test (-1); repo https://github.com/illuin-tech/vidore-benchmark
• ViDoRe-V2 — total ?; splits: test (-1); repo https://github.com/illuin-tech/vidore-benchmark
• JinaVDR-Bench — total ?; splits: test (-1)
• REAL-MM-RAG — total ?; splits: test (-1)
• ViDoSeek — total ?; splits: test (-1)
• MMLongBench-Doc — total ?; splits: test (-1)

Metrics

• nDCG@5 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 5. It measures the quality of the ranked list of retrieved documents by discounting relevance by position, normalized by the ideal DCG.

Input / output format

Input: Visual document images (or pages) and a natural language query.

Output: A ranked list of retrieved document IDs or embeddings.

Scoring recipe

def compute_ndcg_at_k(relevant_docs, retrieved_docs, k=5):
    dcg = 0.0
    for i, doc_id in enumerate(retrieved_docs[:k]):
        if doc_id in relevant_docs:
            dcg += 1.0 / math.log2(i + 2)
    num_relevant = min(len(relevant_docs), k)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(num_relevant))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Compression ratios vary per document for adaptive methods; average compression is reported rather than a fixed per-document ratio.
• Evaluation focuses on offline indexing latency and storage reduction, not retrieval latency.
• Performance is evaluated across multiple base models (ColQwen2.5, ColNomic, Jina-v4), so results are not model-agnostic.

Evidence (verbatim from paper)

Following standard VDR practices, we adopt nDCG@5 as the primary evaluation metric. Our experimental validation is performed on a comprehensive suite of six representative VDR benchmarks, totaling 29 distinct datasets (more details in Appendix [E]). These include ViDoRe-V1 (Faysse et al., [2024]), ViDoRe-V2 (macé2025vidorebenchmarkv2raising), JinaVDR-Bench (Günther et al., [2025]), REAL-MM-RAG (Wasserman et al., [2025]), ViDoSeek (Wang et al., [2025b]), and MMLongBench-Doc (Ma et al., [2024b]).

Citation

@misc{yan2026sculpting,
  title={Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Framework},
  author={Yan et al. (2026)},
  year={2026},
  note={arXiv:2602.19549}
}

• arXiv: 2602.19549