Betterbench Assessment

betterbench-assessment

BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices — Reuel et al. (2024) (arXiv:2411.12990, 2024)

What this evaluates

A meta-evaluation framework that scores AI benchmarks across four lifecycle stages to assess their quality, reproducibility, and usability. It evaluates how well benchmarks are designed, implemented, documented, and maintained for both foundation and non-foundation models.

Datasets

• BetterBench Benchmark Collection — total 24; splits: test (24)

Metrics

• lifecycle_score (primary) — range: [0, 10]
  • Scores are assigned on a 0–10 scale per lifecycle stage (Design, Implementation, Documentation, Maintenance). The Usability score is the weighted average of Implementation, Documentation, and Maintenance. The overall score is the average of the Design score and the Usability score.

Input / output format

Input: Benchmark documentation, code repositories, metadata, and usage guidelines provided by the benchmark creators.

Output: Numerical scores (0–10) for each of the four lifecycle stages, plus a composite Usability score and overall assessment score.

Scoring recipe

def calculate_betterbench_score(benchmark_artifacts):
    scores = {}
    for stage in ['Design', 'Implementation', 'Documentation', 'Maintenance']:
        scores[stage] = sum(criterion_score for criterion in stage_criteria)
    usability = weighted_average(scores['Implementation'], scores['Documentation'], scores['Maintenance'])
    overall = average(scores['Design'], usability)
    return scores, usability, overall

Common pitfalls

• Benchmarks frequently fail to include replication scripts or report statistical significance, heavily penalizing Implementation and Maintenance scores.
• Design scores do not always correlate with usability for non-foundation model benchmarks, indicating design quality alone is insufficient.
• Scores are deterministic per benchmark but require statistical testing across categories to claim significance.

Evidence (verbatim from paper)

Table 1 showcases the average scores per benchmark lifecycle stage, showing that for both FM and non-FM benchmarks, the implementation stage tends to be the weakest area, followed by maintenance. The usability score is the weighted average of the implementation, documentation, and maintenance scores.

Citation

@misc{reuel2024betterbench,
  title={BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices},
  author={Reuel et al. (2024)},
  year={2024},
  note={arXiv:2411.12990}
}

• arXiv: 2411.12990