Tmmluplus Eval

tmmluplus-eval

An Improved Traditional Chinese Evaluation Suite for Foundation Model — Zhi-Rui Tam et al. (2024) (arXiv:2403.01858, 2024)

What this evaluates

Evaluates foundation models' multitask language understanding and reasoning capabilities in Traditional Chinese across diverse academic subjects including STEM, social sciences, humanities, and other domains.

Datasets

• TMMLU+ — total 22690; splits: test (-1), dev (-1)

Metrics

• average accuracy (%) (primary) — range: percent
  • Calculated as the proportion of correctly predicted answers out of the total number of questions, multiplied by 100. For human baselines, it is computed as the passing rate based on the proportion of correct responses to total responses per question.

Input / output format

Input: Multiple-choice questions in Traditional Chinese (or Simplified Chinese for cross-lingual experiments), formatted as zero-shot or five-shot prompts with context examples.

Output: Single letter/option answer extracted via regular expression parsing from the model's raw text response.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Chain-of-thought prompting degrades performance on this benchmark compared to direct answer prompting.
• Model performance is highly sensitive to whether the prompt language (Simplified vs. Traditional Chinese) matches the model's primary training language.
• Tokenizer fertility score strongly correlates with accuracy; models with high fertility (longer token sequences for Chinese characters) underperform.

Evidence (verbatim from paper)

Table 2: Zero-shot/Five-shot average accuracy (%) in an answer-only setting. The average accuracy over the subjects within each category. Responses are parsed using a modified regular expression by Huang et al. (2023) with Traditional Chinese characters added.

Citation

@misc{tam2024tmmluplus,
  title={An Improved Traditional Chinese Evaluation Suite for Foundation Model},
  author={Zhi-Rui Tam et al. (2024)},
  year={2024},
  note={arXiv:2403.01858}
}

• arXiv: 2403.01858