Hkmmlu Eval

hkmmlu-eval

Measuring Hong Kong Massive Multi-Task Language Understanding — Cao et al. (2025) (arXiv:2505.02177, 2025)

What this evaluates

Evaluates large language models' multilingual comprehension of Hong Kong-specific knowledge, Cantonese linguistic capabilities, and reasoning across STEM, social sciences, and humanities in both Traditional and Simplified Chinese.

Datasets

• HKMMLU — total 117248; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multi-choice questions, extracted using regular expressions from model outputs.
• BLEU — range: [0, 1]
  • Standard n-gram overlap metric for machine translation evaluation.
• METEOR — range: [0, 1]
  • Metric based on alignment of words between reference and hypothesis, considering synonyms and stemming.
• ROUGE-L — range: [0, 1]
  • Measures the longest common subsequence between reference and hypothesis translations.

Input / output format

Input: Multi-choice questions with options in Traditional or Simplified Chinese; source sentences in Mandarin or Cantonese for translation tasks.

Output: For multi-choice: a single option letter/answer extracted via regex. For translation: a generated target language sentence.

Scoring recipe

def score_mc(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return correct / len(golds) * 100

def score_translation(predictions, references):
    bleu = compute_bleu(references, predictions)
    meteor = compute_meteor(references, predictions)
    rouge_l = compute_rouge_l(references, predictions)
    return bleu, meteor, rouge_l

Common pitfalls

• Chain-of-thought prompting significantly drops average accuracy across most models, despite improving STEM reasoning.
• Translation evaluation shows severe asymmetry; models consistently perform much worse translating Mandarin to Cantonese than vice versa.
• Few-shot prompting does not guarantee performance gains and can cause sharp accuracy drops for certain model families.

Evidence (verbatim from paper)

We use regular expressions to extract the answers and calculate the percentage of correct answers. For translation tasks, we directly prompt the LLM to translate the sentences and utilize BLEU, METEOR, and ROUGE-L for evaluation.

Citation

@misc{cao2025hkmmlu,
  title={Measuring Hong Kong Massive Multi-Task Language Understanding},
  author={Cao et al. (2025)},
  year={2025},
  note={arXiv:2505.02177}
}

• arXiv: 2505.02177