Rank Distillm Eval

rank-distillm-eval

Rank-DistiLLM: Closing the Effectiveness Gap Between Cross-Encoders and LLMs for Passage Re-Ranking — Schlatt et al. (2024) (arXiv:2405.07920, 2024)

What this evaluates

Probes the re-ranking capability of distilled cross-encoder models on passage retrieval tasks. It evaluates ranking quality on in-domain benchmarks (TREC Deep Learning tracks) and out-of-domain generalization across diverse corpora (TIREx framework), while also measuring computational efficiency.

Datasets

• Rank-DistiLLM — total ?; splits: train (-1), val (-1); repo https://github.com/webis-de/ECIR-25
• TREC DL 2019 — total ?; splits: test (-1)
• TREC DL 2020 — total ?; splits: test (-1)
• TIREx — total ?; splits: test (-1)

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. It measures the quality of the top-10 ranked results against ground-truth relevance judgments by discounting gains logarithmically based on position.

Input / output format

Input: Query (truncated to ≤32 tokens) paired with a candidate list of up to 100 retrieved passages (each truncated to ≤256 tokens).

Output: A ranked list or relevance scores for the candidate passages, ordered by predicted relevance to the query.

Scoring recipe

def compute_ndcg_at_10(pred_scores, gold_labels, k=10):
    import math
    ranked_indices = sorted(range(len(pred_scores)), key=lambda i: pred_scores[i], reverse=True)
    dcg = sum((2**gold_labels[ranked_indices[i]] - 1) / math.log2(i + 2) for i in range(k))
    ideal_labels = sorted(gold_labels, reverse=True)
    idcg = sum((2**ideal_labels[i] - 1) / math.log2(i + 2) for i in range(k))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Models are evaluated on top-100 passages retrieved by specific baselines (BM25 or ColBERTv2), so results are not retrieval-agnostic and depend heavily on the initial retriever.
• Statistical significance is reported using Holm-Bonferroni-corrected t-tests against the best monoELECTRA baseline, not pairwise across all models.
• Efficiency metrics (latency/memory) are hardware-specific (NVIDIA A100 40GB) and measured on a fixed batch/window size, limiting direct cross-system comparisons.

Evidence (verbatim from paper)

Table[1] lists nDCG@10 scores of monoELECTRA, a cross-encoder using ELECTRA*[[10]]* as the backbone encoder, fine-tuned on the data mentioned in Section[3.2], and evaluated on the TREC DL 2019 and 2020 tasks when re-ranking the top 100 passages retrieved by BM25 and ColBERTv2.

Citation

@misc{schlatt2024rankdistillm,
  title={Rank-DistiLLM: Closing the Effectiveness Gap Between Cross-Encoders and LLMs for Passage Re-Ranking},
  author={Schlatt et al. (2024)},
  year={2024},
  note={arXiv:2405.07920}
}

• arXiv: 2405.07920