Coverage At Top K

coverage_at_top_k

A Systematic Analysis of Base Model Choice for Reward Modeling — Ahrabian et al. (2025) (arXiv:2505.10775, 2025)

What this evaluates

Measures the alignment between a proxy benchmark's ranking and a target reward modeling benchmark's ranking at the top-k positions. It quantifies how many of the highest-performing models on a reward benchmark are also identified as top performers on a given proxy benchmark.

Datasets

• Curated Benchmark Suite (33 benchmarks) — total ?; splits: test (-1)

Metrics

• coverage_at_top_k (primary) — range: [0, 1]
  • C(β,ρ,L,k) = |T_β(L,k) ∩ T_ρ(L,k)| / k, where T_x(y,z) returns the top z models in y on benchmark x. It calculates the fraction of overlapping top-k models between a proxy benchmark β and a reward benchmark ρ.
• pearson_correlation — range: [-1, 1]
  • Standard Pearson correlation coefficient measuring linear relationship between benchmark scores and reward modeling performances.
• spearman_correlation — range: [-1, 1]
  • Standard Spearman rank correlation coefficient measuring monotonic relationship between benchmark scores and reward modeling performances.

Input / output format

Input: A set of LLMs L, their scores on a proxy benchmark β, and their scores on a target reward benchmark ρ.

Output: A coverage value between 0 and 1 for a given k, or correlation coefficients.

Scoring recipe

def compute_coverage(proxy_scores, reward_scores, k):
    top_k_proxy = set(m for m, _ in sorted(proxy_scores, key=lambda x: x[1], reverse=True)[:k])
    top_k_reward = set(m for m, _ in sorted(reward_scores, key=lambda x: x[1], reverse=True)[:k])
    return len(top_k_proxy & top_k_reward) / k

Common pitfalls

• Coverage is highly sensitive to k; the paper notes low coverage at higher ranks, making it less effective for broad model selection.
• High correlation does not guarantee high coverage; benchmarks can be strongly correlated with reward scores yet fail to identify top-performing models.

Evidence (verbatim from paper)

Given a benchmark β and reward bench ρ, we formally define the coverage at top-k as C(β,ρ,L,k)=|T_β(L,k)∩T_ρ(L,k)|/k where T_x(y,z) is the top z LLMs in y on benchmark x. To simulate a real-world search where we need high coverage at higher ranks, we filter out any benchmark with less than 0.4 and 0.7 coverage at k=5 and k=10, respectively.

Citation

@misc{ahrabian2025systematic,
  title={A Systematic Analysis of Base Model Choice for Reward Modeling},
  author={Ahrabian et al. (2025)},
  year={2025},
  note={arXiv:2505.10775}
}

• arXiv: 2505.10775