Livebench Eval

livebench-eval

LiveBench: A Challenging, Contamination-Limited LLM Benchmark — White et al. (2024) (arXiv:2406.19314, 2024)

What this evaluates

Evaluates large language models across 18 tasks spanning math, coding, reasoning, language, instruction following, and data analysis. It uses dynamically updated, objectively scored questions from recent real-world sources to minimize test-set contamination and avoid LLM-judging biases.

Datasets

• LiveBench — total ?; splits: test (-1); repo https://github.com/livebench/livebench

Metrics

• LiveBench score (primary) — range: [0, 1]
  • Each question is scored 0 or 1 via automated parsing against ground truth. The task score is the average of question scores, the category score is the average of task scores, and the final LiveBench score is the average of the six category scores.

Input / output format

Input: Single-turn prompt/question per instance, formatted using the model's specific chat template from FastChat.

Output: Model-generated text response, which is parsed automatically to determine a binary 0 or 1 score.

Scoring recipe

question_scores = [parse_and_score(response, ground_truth) for response, ground_truth in zip(predictions, gold)]
task_score = sum(question_scores) / len(question_scores)
category_score = average(task_scores_for_category)
final_score = average(category_scores)

Common pitfalls

• Automated parsing must be carefully validated to avoid unfairly penalizing correct answers due to formatting differences.
• Benchmark scores drop over time as new model versions are released and questions are updated, so cross-time comparisons require matching benchmark versions.
• LLM-judging should be avoided for these tasks as it introduces style and capability biases compared to ground-truth scoring.

Evidence (verbatim from paper)

For all models and tasks, we perform single-turn evaluation with temperature 0, unless otherwise noted in the model card. All models run with their respective templates from our updated version of FastChat... For each question, a model receives a score from 0 to 1. For each model, we compute the score on each task as the average of all questions, we compute the score on each of the six categories as the average of all their tasks, and we compute the final LiveBench score as the average of all six categories.

Citation

@misc{white2024livebench,
  title={LiveBench: A Challenging, Contamination-Limited LLM Benchmark},
  author={White et al. (2024)},
  year={2024},
  note={arXiv:2406.19314}
}

• arXiv: 2406.19314