Seaexam Eval

seaexam-eval

SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asia — Liu et al. (2025) (arXiv:2502.06298, 2025)

What this evaluates

Evaluates LLMs' ability to answer local, culturally grounded multiple-choice questions in Southeast Asian languages (Indonesian, Thai, Vietnamese). It probes regional knowledge, language comprehension, and alignment with actual local usage compared to translated benchmarks.

Datasets

• SeaExam — total ?; splits: test (-1); repo https://github.com/DAMO-NLP-SG/SeaExam

Metrics

• accuracy (%) (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions. Calculated as (number of correct predictions / total number of questions) × 100.

Input / output format

Input: 3-shot prompt containing three example question-answer pairs followed by a multiple-choice question in a Southeast Asian language.

Output: Model selects the correct option letter/answer from the provided choices.

Scoring recipe

predictions = model.generate(input_text)
gold = question['answer']
accuracy = sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold) * 100

Common pitfalls

• Directly comparing SeaExam accuracy with SeaBench judge scores without normalization, as they use fundamentally different scoring mechanisms.
• Assuming translated benchmarks like MMLU-SEA capture the same cultural and linguistic nuances as locally constructed exams.
• Overlooking that MCQ formats may provide linguistic cues that artificially inflate performance compared to open-ended tasks.

Evidence (verbatim from paper)

Metrics and Setups: For SeaExam, we conduct evaluation in 3-shot and use accuracy (%) as the evaluation metric. For SeaBench, we employ LLMs-as-a-JudgeZheng et al. ([2023b]); Bai et al. ([2023]); Ying et al. ([2024]), setting GPT-4o as the judge model to evaluate LLM’s responses based on the reference answers (construction details in Section[2.2]). Considering that different categories of questions focus on assessing different aspects of model performance, we have designed a list of priority evaluation aspects for each category to facilitate a comprehensive judgment. We prompt GPT-4o to rate each response on a scale from 1 to 10.

Citation

@misc{liu2025seaexam,
  title={SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asia},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2502.06298}
}

• arXiv: 2502.06298