League Leaderboard Quality

league-leaderboard-quality

League: Leaderboard Generation on Demand — Jian Wu et al. (2025) (arXiv:2502.18209, 2025)

What this evaluates

Evaluates an LLM-based framework's ability to dynamically generate research leaderboards by measuring topic relevance, content quality (coverage, recency, structure), and generation speed compared to manual curation.

Datasets

• arXiv papers on specific research topics — total ?; splits: test (-1)

Metrics

• Topic-related Recall — range: [0, 1]
  • Percentage of retrieved papers that are correctly identified as relevant to the target research topic.
• Topic-related Precision — range: [0, 1]
  • Percentage of retrieved papers that are correctly identified as relevant, filtering out irrelevant items.
• Leaderboard Content Quality (primary) — range: [1, 5]
  • LLM-judged score on a 5-point scale for Coverage, Latest, and Structure. Multiaspect is the average of these three scores. Calibrated by human experts.
• Construction Speed — range: seconds
  • Total time in seconds to generate the leaderboard, calculated as the sum of API invocation times for League, compared to manual time components.
• Pearson Correlation Coefficient — range: [-1, 1]
  • Measures linear correlation between LLM-generated quality scores and human expert rankings of leaderboard pairs.
• Table Classification F1 — range: [0, 1]
  • Harmonic mean of precision and recall for classifying tables into main results, ablation studies, or others.
• Table NER F1 — range: [0, 1]
  • Harmonic mean of precision and recall for extracting methods, datasets, experimental settings, and metrics from tables.

Input / output format

Input: Crawled research papers (abstracts and tables) for a given research topic.

Output: A structured leaderboard listing papers, models, datasets, experimental settings, and results, typically containing 5, 10, 15, or 20 items.

Scoring recipe

def score_leaderboard(retrieved_papers, ground_truth_papers, generated_leaderboard, api_times, human_rankings):
    relevant_count = sum(1 for p in retrieved_papers if is_relevant(p, ground_truth_papers))
    recall = relevant_count / len(retrieved_papers)
    precision = relevant_count / len(ground_truth_papers)
    
    cov = llm_judge(generated_leaderboard, 'Coverage', scale=5)
    lat = llm_judge(generated_leaderboard, 'Latest', scale=5)
    stru = llm_judge(generated_leaderboard, 'Structure', scale=5)
    multiaspect = (cov + lat + stru) / 3
    
    speed_league = sum(api_times)
    speed_manual = T_r + T_b + T_f + T_e + T_c
    
    pearson_corr = pearsonr(llm_scores, human_rankings)
    
    return recall, precision, cov, lat, stru, multiaspect, speed_league, speed_manual, pearson_corr

Common pitfalls

• Manual construction time is estimated via component breakdown rather than direct stopwatch measurement, introducing approximation error.
• LLM-based content quality scores are subjective and depend heavily on the specific prompt and calibration method used.
• Topic-related recall/precision relies on an initial regex-based arXiv crawler, which may miss relevant papers or include noise, affecting downstream leaderboard quality.

Evidence (verbatim from paper)

We evaluate the quality of content from the following two aspects. (i) Recall: It measures whether all items in the generated leaderboard are related to the given research topic. (ii) Precision: It identifies irrelevant items, ensuring that the items in the leaderboards are pertinent and directly support the given research topic. The evaluation metric of leaderboard Content Quality includes four aspects. Each aspect is judged by LLMs according to a 5-point, calibrated by human experts.

Citation

@misc{wu2025league,
  title={League: Leaderboard Generation on Demand},
  author={Jian Wu et al. (2025)},
  year={2025},
  note={arXiv:2502.18209}
}

• arXiv: 2502.18209