Qwen2 Eval

qwen2-eval

Qwen2 Technical Report — An Yang et al. (2024) (arXiv:2407.10671, 2024)

What this evaluates

This protocol evaluates large language models across core competencies including general knowledge, reasoning, coding, and mathematics. It also assesses multilingual understanding, instruction following, and alignment with human preferences.

Datasets

• MMLU — total ?; splits: test (-1)
• MMLU-Pro — total ?; splits: test (-1)
• GPQA — total ?; splits: test (-1)
• HumanEval — total ?; splits: test (-1)
• GSM8K — total ?; splits: test (-1)
• MT-Bench — total ?; splits: test (-1)
• IFEval — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions or exact-match for generation tasks. Calculated as (number of correct predictions / total instances) * 100.
• MT-Bench score — range: 1-10
  • Average preference score assigned by human evaluators or LLM judges on a scale of 1 to 10.

Input / output format

Input: Text prompts formatted with few-shot or zero-shot examples for base models, or direct instruction prompts for instruction-tuned models.

Output: Model-generated text responses, including multiple-choice selections, code snippets, mathematical derivations, or free-form answers.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if normalize(p) == normalize(g))
    return (correct / len(gold)) * 100

def compute_mt_bench(predictions, gold):
    scores = [score_response(p, g) for p, g in zip(predictions, gold)]
    return sum(scores) / len(scores)

Common pitfalls

• Shot counts vary widely across benchmarks (0-shot to 25-shot), which can drastically change performance and comparability.
• Coding benchmarks like HumanEval and MBPP use pass@1 or exact match, but EvalPlus applies stricter execution filters that may penalize valid but non-standard code.
• MT-Bench and Arena-Hard rely on LLM-as-a-judge or human preference, which can be biased by response length or formatting.

Evidence (verbatim from paper)

Qwen2-72B outperforms Llama-3-70B in general knowledge understanding on both MMLU and MMLU-Pro, achieving accuracy improvements of 4.7 and 2.8, respectively. In scientific assessments, Qwen2-72B demonstrates superiority over Llama-3-70B with enhancements of 1.6 and 9.8 on GPQA and Theorem QA.

Citation

@misc{yang2024qwen2,
  title={Qwen2 Technical Report},
  author={An Yang et al. (2024)},
  year={2024},
  note={arXiv:2407.10671}
}

• arXiv: 2407.10671