Syndl Passage Retrieval Eval

syndl-passage-retrieval-eval

SynDL: A Large-Scale Synthetic Test Collection for Passage Retrieval — Rahmani et al. (2024) (arXiv:2408.16312, 2024)

What this evaluates

Evaluates the quality and alignment of a synthetic passage retrieval test collection (SynDL) by measuring how well system rankings on synthetic relevance judgments match those from official human-annotated TREC Deep Learning Track collections. It probes whether LLM-generated judgments can reliably substitute human assessors for deep relevance evaluation and system ranking.

Datasets

• SynDL — total ?; splits: test (-1); repo https://github.com/rahmanidashti/

Metrics

• Kendall rank correlation coefficient ($\tau$) (primary) — range: [-1, 1]
  • Measures the rank agreement between system rankings computed on the synthetic test collection versus the official human-annotated TREC DL collection. Computed over system-level NDCG scores. Higher values indicate better alignment with human judgments.
• NDCG@10 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10, used to compute system rankings for correlation analysis.
• NDCG@100 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 100, used to compute system rankings for correlation analysis.

Input / output format

Input: System retrieval rankings (ordered lists of passages) for queries from the TREC Deep Learning Track, evaluated against both official human relevance judgments and synthetic LLM-generated relevance judgments.

Output: Kendall's $\tau$ correlation coefficient between human and synthetic system rankings, along with NDCG@10 and NDCG@100 scores per system on the synthetic collection.

Scoring recipe

def evaluate_collection(system_runs, human_judgments, synthetic_judgments):
    human_scores = []
    synthetic_scores = []
    for run in system_runs:
        h_score = ndcg_at_k(run.ranks, human_judgments, k=10)
        s_score = ndcg_at_k(run.ranks, synthetic_judgments, k=10)
        human_scores.append(h_score)
        synthetic_scores.append(s_score)
    tau = kendall_tau(human_scores, synthetic_scores)
    return tau

Common pitfalls

• The paper evaluates the test collection's quality via system-level rank correlation, not just raw system performance on the synthetic data.
• Readers may assume LLM judgments inherently bias rankings toward systems using the same LLM; the protocol explicitly tests and controls for this by categorizing systems (GPT, T5, etc.).
• Correlation is computed at the system level using NDCG scores, not at the query or passage level.

Evidence (verbatim from paper)

To effectively evaluate our SynDL test collection, we follow the evaluation setups in (Faggioli et al., [2023]; Rahmani et al., [2024a]), which use the correlation test on the system ranking when evaluated using human judgments and LLM judgments. ... Figure[1] shows the evaluated correlation via Kendall rank correlation coefficients when evaluated with NDCG in two depths (@10 and @100).

Citation

@misc{rahmani2024syndl,
  title={SynDL: A Large-Scale Synthetic Test Collection for Passage Retrieval},
  author={Rahmani et al. (2024)},
  year={2024},
  note={arXiv:2408.16312}
}

• arXiv: 2408.16312