Coranking Eval

coranking-eval

CoRanking: Collaborative Ranking with Small and Large Ranking Agents — Wenhan Liu et al. (arXiv:2503.23427, 2025)

What this evaluates

Evaluates a collaborative reranking framework that combines a small efficient reranker with a large LLM-based reranker. It uses a reinforcement learning-trained passage order adjuster to mitigate positional bias and reduce latency while maintaining ranking effectiveness on standard IR benchmarks.

Datasets

• TREC DL (DL19, DL20) — total ?; splits: test (-1)
• BEIR (TREC-Covid, Robust04, Trec-News) — total ?; splits: test (-1)
• BRIGHT (Economics, Earth Science, Robotics) — total ?; splits: test (-1)

Metrics

• NDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Computes the sum of graded relevance values of results up to position 10, discounted logarithmically by position, and normalizes by the ideal DCG@10.

Input / output format

Input: A query and a list of candidate passages (typically top-20 or all retrieved passages depending on the reranking stage).

Output: A re-ranked list of passages ordered by predicted relevance.

Scoring recipe

def ndcg_at_10(predictions, gold):
    # predictions: list of predicted ranks or scores
    # gold: list of ground truth relevance labels
    # Compute DCG@10
    dcg = sum(rel / log2(i + 2) for i, rel in enumerate(gold[:10]))
    # Compute IDCG@10 (ideal)
    ideal_gold = sorted(gold, reverse=True)[:10]
    idcg = sum(rel / log2(i + 2) for i, rel in enumerate(ideal_gold))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Confusing the naive collaborative baseline (SLR+LLR) with the proposed CoRanking framework, which includes a dedicated POA module for order alignment.
• Assuming NDCG@10 is calculated on the full passage list rather than strictly the top-10 reranked results.
• Overlooking that reported latency varies significantly across datasets due to differing passage lengths, not just model architecture.

Evidence (verbatim from paper)

We evaluate our framework on three established information retrieval benchmarks: TREC DL, BEIR, and BRIGHT... Table 1: Results (NDCG@10) on TREC, BEIR, and BRIGHT benchmarks. The “Avg.” represents the averaged result of all 8 datasets.

Citation

@misc{liu2025coranking,
  title={CoRanking: Collaborative Ranking with Small and Large Ranking Agents},
  author={Wenhan Liu et al.},
  year={2025},
  note={arXiv:2503.23427}
}

• arXiv: 2503.23427