Benchmark Diversity Stability Eval

benchmark-diversity-stability-eval

Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarks — Zhang et al. (2024) (arXiv:2405.01719, 2024)

What this evaluates

Evaluates the inherent trade-off between diversity (agreement of model rankings across tasks) and stability (sensitivity of final rankings to label noise) in multi-task machine learning benchmarks. It quantifies how much a benchmark's leaderboard ranking changes when trivial label noise is injected, and how diverse the rankings are across its constituent tasks.

Datasets

• GLUE — total ?; splits: test (-1)
• SuperGLUE — total ?; splits: test (-1)
• MTEB — total ?; splits: test (-1)
• BigBenchHard — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)
• OpenLLM — total ?; splits: test (-1)
• VTAB — total ?; splits: test (-1)
• ImageNet — total ?; splits: val (-1)

Metrics

• Kendall's W — range: [0, 1]
  • Measures diversity as the degree of agreement among model rankings across different tasks within a benchmark. Computed using the Kendall's W coefficient.
• Kendall's τ (primary) — range: [-1, 1]
  • Measures sensitivity (ranking change) after injecting label noise. Computed as the Kendall rank correlation coefficient between the original leaderboard ranking and the ranking after noise injection.
• MRC — range: [0, 1]
  • Mean Rank Change, quantifying the average absolute shift in model positions after noise injection.

Input / output format

Input: Model performance scores or aggregated rankings across multiple tasks within a benchmark.

Output: Diversity score (Kendall's W) and sensitivity scores (Kendall's τ, MRC) quantifying how much the final model ranking changes under label noise.

Scoring recipe

# 1. Compute diversity across tasks
diversity = kendalls_w(task_rankings)

# 2. Determine noise preservation ratio
epsilon = min(0.01, std_min / std_max)

# 3. Inject label noise preserving epsilon fraction
noisy_scores = inject_label_noise(original_scores, preserve_fraction=epsilon)

# 4. Compute sensitivity via ranking changes
noisy_rankings = compute_rankings(noisy_scores)
sensitivity_tau = kendalls_tau(original_rankings, noisy_rankings)
sensitivity_mrc = mean_rank_change(original_rankings, noisy_rankings)

Common pitfalls

• Sensitivity depends on a dynamic epsilon set by task standard deviations, not a fixed noise level, which can disproportionately affect benchmarks with high variance tasks.
• For OpenLLM and MTEB, only the top-100 models are evaluated to mitigate noise, unlike other benchmarks that use all leaderboard candidates, making cross-benchmark comparisons sensitive to candidate pool size.
• Rankings are derived from aggregated scores across tasks, so the evaluation is highly sensitive to how individual task scores are normalized or weighted before aggregation.

Evidence (verbatim from paper)

Diversity and sensitivity scores are computed for each benchmark based on equation 3 and Algorithm 1. For both measures, all models are used for calculation, i.e., λ contains all models in the leaderboard. The only exceptions are OpenLLM and MTEB, where we focus on the top-100 models out of thousands of candidates to mitigate the influence of less reliable ones. ... The ranking change is measured by both Kendall's τ (top) and MRC (bottom).

Citation

@misc{zhang2024diversitystability,
  title={Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarks},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2405.01719}
}

• arXiv: 2405.01719