Seabench Eval

seabench-eval

SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asia — Liu et al. (2025) (arXiv:2502.06298, 2025)

What this evaluates

Evaluates LLMs' ability to handle open-ended, daily interaction scenarios in Southeast Asian languages. It probes contextual adaptation, instruction following, and safety in real-world multilingual usage.

Datasets

• SeaBench — total ?; splits: test (-1); repo https://github.com/DAMO-NLP-SG/SeaExam

Metrics

• LLM-as-a-Judge Score (primary) — range: [1, 10]
  • Average score from 1 to 10 assigned by GPT-4o acting as a judge. Responses are evaluated against reference answers using category-specific priority aspects.

Input / output format

Input: Zero-shot prompt containing an open-ended question or multi-turn conversation scenario in a Southeast Asian language.

Output: Model generates a free-form text response.

Scoring recipe

responses = model.generate(input_text)
judge = GPT-4o
scores = [judge.score(response, reference, criteria) for response, reference in zip(responses, references)]
avg_score = sum(scores) / len(scores)

Common pitfalls

• Using translated benchmarks (MT-bench-SEA) as a direct proxy for local daily usage, which fails to capture region-specific safety and contextual nuances.
• Comparing open-ended judge scores directly with MCQ accuracy without converting to a common scale (e.g., high-score rate or full mark rate).
• Assuming safety performance aligns with primary language alignment, as multilingual safety often suffers due to training data bias.

Evidence (verbatim from paper)

Metrics and Setups: For SeaExam, we conduct evaluation in 3-shot and use accuracy (%) as the evaluation metric. For SeaBench, we employ LLMs-as-a-JudgeZheng et al. ([2023b]); Bai et al. ([2023]); Ying et al. ([2024]), setting GPT-4o as the judge model to evaluate LLM’s responses based on the reference answers (construction details in Section[2.2]). Considering that different categories of questions focus on assessing different aspects of model performance, we have designed a list of priority evaluation aspects for each category to facilitate a comprehensive judgment. We prompt GPT-4o to rate each response on a scale from 1 to 10.

Citation

@misc{liu2025seaexam,
  title={SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asia},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2502.06298}
}

• arXiv: 2502.06298