Mmlu Eval

mmlu-eval

GEM: Empowering LLM for both Embedding Generation and Language Understanding — Caojin Zhang et al. (2025) (arXiv:2506.04344, 2025)

What this evaluates

Evaluates broad language understanding and reasoning capabilities across multiple academic and professional domains using multiple-choice questions. It tests the model's ability to process and answer questions in a few-shot setting.

Datasets

• MMLU — total ?; splits: test (-1); HF cais/mmlu

Metrics

• macro_avg/acc_char (primary) — range: percent
  • Macro-averaged character-level accuracy across all subjects. Answers are evaluated using character-level similarity rather than exact string matching to handle formatting variations.

Input / output format

Input: Multiple-choice questions with a 5-shot context window.

Output: Selected option letter or generated text answer.

Scoring recipe

# Follow Llama 3.1 model card protocol
predictions = []
for question in mmlu_test:
    prompt = build_5shot_prompt(question)
    answer = model.generate(prompt)
    predictions.append(extract_option(answer))
acc_char = character_accuracy(predictions, gold_labels)
macro_acc = mean(acc_char_per_subject)
return macro_acc

Common pitfalls

• The paper reports 'acc_char' (character accuracy) instead of exact-match accuracy, which can yield different scores for synonymous or formatted answers.
• The paper incorrectly states MMLU covers '57 languages'; it actually covers 57 academic/professional subjects, which may confuse evaluators expecting language-specific splits.

Evidence (verbatim from paper)

For MMLU, we follow Llama 3.1 model card and use five-shot and report macro_avg/acc_char.

Citation

@misc{zhang2025gem,
  title={GEM: Empowering LLM for both Embedding Generation and Language Understanding},
  author={Caojin Zhang et al. (2025)},
  year={2025},
  note={arXiv:2506.04344}
}

• arXiv: 2506.04344