Sdsko Pub Vdr Eval

sdsko-pub-vdr-eval

SDS KoPub VDR: A Benchmark Dataset for Visual Document Retrieval in Korean Public Documents — Jaehoon Lee et al. (2025) (arXiv:2511.04910, 2025)

What this evaluates

Evaluates a model's ability to retrieve relevant Korean public document pages given a text query, comparing text-only parsing against multimodal visual understanding. It probes cross-modal reasoning, layout awareness, and the capacity to interpret tables, charts, and complex visual structures in administrative documents.

Datasets

• SDS KoPub VDR — total 600; splits: test (600)

Metrics

• Recall@k (primary) — range: [0, 1]
  • Binary indicator of whether the ground-truth page appears within the top-k retrieved results. Averaged over all queries. Evaluated at k=1, 3, 5, 10.
• nDCG@k — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank k. For single-ground-truth queries, it measures how early the correct page appears in the ranked list, discounting relevance by log2(rank+1). Evaluated at k=1, 3, 5, 10.

Input / output format

Input: Task 1: Text query + PDF-extracted text of a candidate page. Task 2: Text query + full-page image of a candidate page.

Output: Ranked list of top-10 candidate pages sorted by cosine similarity to the query embedding.

Scoring recipe

import math
def score(predictions, golds, k=10):
    recalls, ndcgs = [], []
    for pred, gold in zip(predictions, golds):
        rank = pred.index(gold) + 1 if gold in pred else len(pred) + 1
        recalls.append(1.0 if rank <= k else 0.0)
        dcg = 1.0 / math.log2(rank + 1) if rank <= k else 0.0
        idcg = 1.0 / math.log2(2)
        ndcgs.append(dcg / idcg)
    return sum(recalls)/len(recalls), sum(ndcgs)/len(ndcgs)

Common pitfalls

• Relying solely on PDF-extracted text for Task 1 leads to severe performance degradation on pages with tables, charts, or complex layouts due to parsing errors.
• Using non-native embedding dimensions or applying projection layers instead of the model's native configuration introduces memory-based bias and breaks comparability.
• Confusing the input modality: Task 1 requires text-only inputs, while Task 2 requires full-page images; swapping them invalidates the modality-specific evaluation.

Evidence (verbatim from paper)

The performance of each retrieval model is evaluated using two standard metrics widely adopted in information-retrieval research: Recall and Normalized Discounted Cumulative Gain (nDCG). Recall@k measures whether the relevant document appears within the top-k retrieved results, thereby assessing the model's coverage of correct answers. It serves as an indicator of whether the retrieval system successfully includes the ground-truth document among its top-ranked candidates. nDCG@k evaluates the ranking quality of the retrieved list by assigning higher scores when the correct document is ranked closer to the top.

Citation

@misc{lee2025sdsko-pub-vdr,
  title={SDS KoPub VDR: A Benchmark Dataset for Visual Document Retrieval in Korean Public Documents},
  author={Jaehoon Lee et al. (2025)},
  year={2025},
  note={arXiv:2511.04910}
}

• arXiv: 2511.04910