Wizardlm Eval

wizardlm-eval

WizardLM: Empowering large pre-trained language models to follow complex instructions — Xu et al. (2023) (arXiv:2304.12244, 2023)

What this evaluates

Probes instruction-following capability on complex, real-world prompts across diverse domains like coding, math, reasoning, and formatting. It measures how well models handle demanding, multi-step tasks compared to baselines through blind pairwise human comparison.

Datasets

• WizardEval — total 218; splits: test (218)

Metrics

• win_rate (primary) — range: percent
  • Calculated from pairwise comparisons by comparing the frequency of wins, losses, and ties between each pair of models. Responses are ranked 1-5 (1 best) across five criteria: Relevance, Knowledgeable, Reasoning, Calculation, and Accuracy.

Input / output format

Input: A single open-domain instruction from real-world sources (e.g., GitHub, ShareGPT, Twitter, Reddit, Discord) covering one of 29 skills/domains.

Output: A natural language response generated by the model to fulfill the instruction.

Scoring recipe

for each annotator:
  for each prompt in WizardEval:
    present 4 shuffled model responses
    score each on Relevance, Knowledgeable, Reasoning, Calculation, Accuracy
    rank responses 1-5 (1 best, ties allowed)
aggregate ranks across annotators to compute win/loss/tie frequencies per model pair
win_rate = wins / (wins + losses + ties)

Common pitfalls

• Annotators are explicitly allowed to assign equal scores for comparable instances, which complicates strict ranking and win-rate calculation.
• The evaluation uses a blind pairwise setup with only 10 annotators, so results may vary significantly with different annotator pools.
• WizardEval covers 29 diverse skills, but the paper does not report per-skill breakdowns in the main results table.

Evidence (verbatim from paper)

To evaluate WizardLM, we conduct human evaluation on our crafted testbed WizardEval, which includes 218 real-world human instructions from diverse sources such as online opensource projects (Github, ShareGPT), platforms (Twitter), and forums (Reddit, Discord). ... We perform a blind pairwise comparison between WizardLM-13b and baselines. Specifically, we recruit 10 well-educated annotators. ... The annotators then judge which response is better following criterion ... (1) Relevance, (2) Knowledgeable, (3) Reasoning, (4) Calculation, and (5) Accuracy. Then they should rank the four responses from 1 to 5 (1 means best), and allowing equal scores for comparable instances. To estimate the win rate, we compare the frequency of win, lost, and tie between each pair of models.

Citation

@misc{xu2023wizardlm,
  title={WizardLM: Empowering large pre-trained language models to follow complex instructions},
  author={Xu et al. (2023)},
  year={2023},
  note={arXiv:2304.12244}
}

• arXiv: 2304.12244