Sea Helm Eval

sea-helm-eval

SiamGPT: Quality-First Fine-Tuning for Stable Thai Text Generation — Pairatsuppawat et al. (2025) (arXiv:2512.19455, 2025)

What this evaluates

Evaluates Thai language models across eight competencies including instruction following, multi-turn dialogue stability, natural language understanding, generation, reasoning, safety, and code-switching resistance.

Datasets

• SEA-HELM — total ?; splits: test (-1)

Metrics

• SEA-HELM Average Score (primary) — range: percent
  • Arithmetic mean of normalized scores (0–100 scale) across eight sub-benchmarks: SEA-IFEval, SEA-MTBench, NLG, NLU, NLR, Safety, ThaiExam, and Code Switching. Sub-benchmarks follow their respective standard evaluation protocols.

Input / output format

Input: Instruction prompts, multi-turn dialogue histories, QA pairs, translation/summarization inputs, NLI/causal reasoning statements, and safety evaluation queries in Thai and English.

Output: Model-generated text responses corresponding to each input prompt or task.

Scoring recipe

scores = {}
for benchmark in ["SEA-IFEval", "SEA-MTBench", "NLG", "NLU", "NLR", "Safety", "ThaiExam", "Code Switching"]:
    scores[benchmark] = compute_benchmark_score(predictions[benchmark], gold[benchmark])
final_score = sum(scores.values()) / len(scores)
return final_score

Common pitfalls

• The model's lower NLG score is an intentional design trade-off prioritizing stability and formatting control over open-ended generative fluency.
• Multi-turn dialogue and safety scores are highly sensitive to context consistency and constraint adherence, which may vary significantly across different base models.

Evidence (verbatim from paper)

We report the performance of SiamGPT-32B on the SEA-HELM benchmark suite following the evaluation protocol described in Section[3.4], with the main results summarized in Figure[1]. ... Performance on SEA-MTBench increases from 57.94 to 75.81, indicating that SystemChat-2.0 supervision substantially improves contextual consistency across turns. Instruction following also improves markedly, with SEA-IFEval increasing from 75.47 to 83.00. Overall, the average score across benchmarks increases from 68.46 to 76.03, confirming that Quality-First fine-tuning yields consistent improvements across stability, instruction following, dialogue robustness, and language understanding.

Citation

@misc{pairatsuppawat2025siamgpt,
  title={SiamGPT: Quality-First Fine-Tuning for Stable Thai Text Generation},
  author={Pairatsuppawat et al. (2025)},
  year={2025},
  note={arXiv:2512.19455}
}

• arXiv: 2512.19455