Kmmlu Eval

kmmlu-eval

KMMLU: Measuring Massive Multitask Language Understanding in Korean — Son et al. (2024) (arXiv:2402.11548, 2024)

What this evaluates

This benchmark evaluates large language models' ability to understand and answer expert-level multiple-choice questions in Korean across diverse academic domains. It specifically probes cultural and linguistic alignment, testing whether models can handle native-language nuances and localized knowledge without relying on translated or English-centric training data.

Datasets

• KMMLU — total 35030; splits: test (-1); repo https://github.com/EleutherAI/lm-evaluation-harness

Metrics

• accuracy (primary) — range: percent
  • Calculated as the number of correctly predicted options divided by the total number of questions. The final reported score is the macro-average of accuracy across all subject categories. The multiple-choice format has 4 options, yielding a 25% random baseline.

Input / output format

Input: Korean multiple-choice questions with 4 options. Each instance is presented in a 5-shot few-shot setting, where 5 exemplar questions and answers are provided as context before the target question.

Output: Direct method: The model generates a single option letter or text via greedy decoding. CoT method: The model generates free-form reasoning text followed by the final answer, which is extracted using RegEx.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        # Extract answer: for Direct, use raw output; for CoT, apply RegEx
        pred_answer = extract_answer(pred)
        if pred_answer == gold:
            correct += 1
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Using sampling instead of greedy decoding alters results, as the protocol mandates greedy decoding for both Direct and CoT settings.
• RegEx parsing for CoT outputs is fragile; models that do not strictly follow the expected answer format will yield missing extractions.
• Reporting micro-average instead of macro-average across subjects will skew results toward larger categories, contrary to the paper's methodology.

Evidence (verbatim from paper)

Average accuracy(%) calculated using the Direct method in a 5-shot setting across the entire test set. We report the macro-average accuracy across subjects within each category. Random guessing has an accuracy of 25% on all subjects.

Citation

@misc{son2024kmmlu,
  title={KMMLU: Measuring Massive Multitask Language Understanding in Korean},
  author={Son et al. (2024)},
  year={2024},
  note={arXiv:2402.11548}
}

• arXiv: 2402.11548