Mixeval Eval

mixeval-eval

MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures — Ni et al. (2024) (arXiv:2406.06565, 2024)

What this evaluates

Evaluates LLMs on a dynamically mixed benchmark of real-world web-mined queries and existing datasets to measure alignment with human preferences. It probes a model's general capability, reasoning, and instruction-following across diverse domains, correlating performance with Chatbot Arena Elo scores.

Datasets

• MixEval — total ?; splits: test (-1)
• MixEval-Hard — total ?; splits: test (-1)

Metrics

• Spearman's ranking correlation (primary) — range: [-1, 1]
  • Spearman's rank correlation coefficient measuring the monotonic relationship between a model's score on the benchmark and its Chatbot Arena Elo rating. Computed as the rank correlation between the two score vectors.

Input / output format

Input: Prompted with benchmark questions using official chat templates or FastChat templates. Base models receive a 5-shot in-context learning setup with examples.

Output: Free-form text responses or multiple-choice selections. Responses are parsed by GPT-3.5-Turbo-0125 to extract scores or choices.

Scoring recipe

predictions = []
for question in benchmark:
    response = model.generate(question)
    if question.type == "free-form":
        score = gpt35_parser.score(response)
    else:
        choice = gpt35_parser.extract_choice(response)
        score = 1 if choice == question.gold else 0
    predictions.append(score)
arena_elo_scores = get_arena_elo(model_name)
correlation = spearmanr(predictions, arena_elo_scores)
return correlation

Common pitfalls

• Using rule-based parsing instead of LLM-based parsing (GPT-3.5-Turbo-0125) leads to significant inaccuracies in scoring free-form or multiple-choice answers.
• Assuming the benchmark correlation with Arena Elo provides an exact equivalence; the paper notes the estimation remains approximate due to outliers.
• Evaluating base models without the specified 5-shot setting or chat models without official/FastChat templates, which breaks consistency with the reported results.

Evidence (verbatim from paper)

ρ and e denote the Spearman's ranking correlation and the root mean square error of the linear fit respectively.

Citation

@misc{ni2024mixeval,
  title={MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures},
  author={Ni et al. (2024)},
  year={2024},
  note={arXiv:2406.06565}
}

• arXiv: 2406.06565