qhjqhj00 open-llm-leaderboard-eval
Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena — Myrzakhan et al. (2024) (arXiv:2406.07545, 2024)
What this evaluates
This benchmark evaluates large language models on their ability to answer open-style questions across diverse knowledge and reasoning domains. It specifically probes whether models rely on selection bias and random guessing in multiple-choice formats versus genuinely understanding and generating correct open-ended responses.
Datasets
- MMLU — total ?; splits: test (-1)
- ARC — total ?; splits: test (-1)
- MedMCQA — total ?; splits: test (-1)
- PIQA — total ?; splits: test (-1)
- CommonsenseQA — total ?; splits: test (-1)
- OpenBookQA — total ?; splits: test (-1)
- RACE — total ?; splits: test (-1)
- WinoGrande — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Percentage of correctly answered questions. For MCQ, it is the proportion of responses matching the ground truth ID. For OSQ, it is the proportion of responses judged correct by a GPT-4-0125-preview judge against the ground truth answer.
Input / output format
Input: Dataset questions formatted as either multiple-choice (MCQ) with options or open-style questions (OSQ) requiring free-text generation. Prompts are provided in Appendix D.
Output: For MCQ: a single choice ID or letter. For OSQ: a free-text answer string.
Scoring recipe
def compute_accuracy(predictions, gold, mode='osq'):
correct = 0
for pred, g in zip(predictions, gold):
if mode == 'mcq':
if pred == g: correct += 1
else:
judge_out = llm_judge(prompt=f'Q: {g} A: {pred}')
if judge_out == 'correct': correct += 1
return (correct / len(predictions)) * 100
Common pitfalls
- HellaSwag contains multiple plausible continuations, making single-ground-truth evaluation unreliable; the authors explicitly omit it from the final leaderboard.
- Open-style questions naturally yield lower accuracy (~25% drop) than MCQ due to the absence of selection bias, which should not be conflated with model incompetence.
- LLM-as-a-judge evaluation for OSQ depends heavily on the specific GPT-4 prompt and may still struggle with semantically equivalent but differently phrased correct answers.
Evidence (verbatim from paper)
The results in Table [4] and Figure [4] are based on filtered questions. They show that every model experiences a significant drop in the accuracy for OSQ compared to MCQ. On average, the accuracy of OSQ is lower than MCQ by about 25% for all models. This result can correlate with our concern that the model will “randomly guess” to correct choices but it cannot answer.
Citation
@misc{myrzakhan2024openllmleaderboard,
title={Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena},
author={Myrzakhan et al. (2024)},
year={2024},
note={arXiv:2406.07545}
}
- arXiv: 2406.07545