qhjqhj00 cmmlu-eval
CMMLU: Measuring massive multitask language understanding in Chinese — Li et al. (2023) (arXiv:2306.09212, 2023)
What this evaluates
Evaluates large language models' Chinese language understanding and multitask knowledge across 67 subjects spanning STEM, humanities, social sciences, and China-specific domains. It probes memorization, reasoning, and instruction-following capabilities in a multiple-choice question-answering format.
Datasets
- CMMLU — total ?; splits: test (-1); repo https://github.com/haonan-li/CMMLU
Metrics
macro average accuracy(primary) — range: [0, 1]- Proportion of correctly predicted multiple-choice answers. Reported as a macro average over subjects within each category, and overall macro average across all 67 subjects.
Input / output format
Input: A Chinese multiple-choice question with four options (A, B, C, D), optionally preceded by up to 5 demonstration examples. The prompt begins with '以下是关于[主题]的单项选择题，请直接给出正确答案的选项' and ends with '答案是：'.
Output: For open-source models: the single token with the highest logit probability among 'A', 'B', 'C', 'D'. For commercial models: free-form text parsed via regex to extract the option letter.
Scoring recipe
def compute_accuracy(predictions, golds):
correct = 0
for pred, gold in zip(predictions, golds):
if pred == gold:
correct += 1
return correct / len(golds)
Common pitfalls
- Chain-of-thought prompts often cause models to output answer content instead of the option letter, breaking regex matching and artificially lowering scores.
- Free generation and next-token prediction strategies yield different results; next-token prediction is more efficient but requires careful logit handling, while free generation relies on regex that can fail on complex or wrapped outputs.
Evidence (verbatim from paper)
Our goal is to assess the LLMs performance on CMMLU, which contains multiple-choice questions with one correct answer for each question. ... We report macro average accuracy over subjects within each category. "Overall" = macro average score over all subjects.
Citation
@misc{li2023cmmlu,
title={CMMLU: Measuring massive multitask language understanding in Chinese},
author={Li et al. (2023)},
year={2023},
note={arXiv:2306.09212}
}
- arXiv: 2306.09212