Laobench Eval

laobench-eval

LaoBench: A Large-Scale Multidimensional Lao Benchmark for Large Language Models — Gao et al. (2025) (arXiv:2511.11334, 2025)

What this evaluates

Evaluates large language models on their proficiency in Lao, a low-resource Southeast Asian language. It probes factual knowledge, K12 curriculum alignment, culturally grounded reasoning, bilingual translation fidelity, and open-ended generation quality through multiple-choice, translation, and pairwise arena tasks.

Datasets

• LaoBench — total ?; splits: Lao-7k (7000), Lao-500 (500)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly answered multiple-choice questions. Model outputs are normalized to a single option label (A/B/C/D); unparseable or multiple labels count as incorrect.
• BLEU — range: [0, 1]
  • Corpus-level BLEU score computed with SacreBLEU using Lao-aware tokenization to handle the scriptio continua style without explicit word boundaries.
• Win-rate — range: percent
  • Pairwise win rate against a fixed baseline (GPT-5-High). Ties are treated as half-wins. Scores are averaged across prompts and aggregated over two judge models, reported as a percentage.

Input / output format

Input: For multiple-choice: question stem and four options in Lao. For translation: source text in Lao. For open-ended: instruction prompt in Lao.

Output: For multiple-choice: exactly one option label (A, B, C, or D). For translation/open-ended: generated text in Lao.

Scoring recipe

import re, sacrebleu

# 1. Accuracy (Lao-7k MC)
def normalize_mc(output):
    match = re.search(r'[A-D]', output)
    return match.group(0) if match else None
acc = sum(1 for p, g in zip(preds, golds) if normalize_mc(p) == g) / len(golds)

# 2. BLEU (Lao-7k Translation)
bleu = sacrebleu.corpus_bleu(preds, [golds], tokenize='lao')

# 3. Win-rate (Lao-500 Arena)
wins = []
for prompt in prompts:
    s1 = judge1.compare(prompt, baseline) # 1=win, 0.5=tie, 0=loss
    s2 = judge2.compare(prompt, baseline)
    wins.append((s1 + s2) / 2)
win_rate = (sum(wins) / len(wins)) * 100

Common pitfalls

• Lao is written in a scriptio continua style without explicit word boundaries, so standard BLEU tokenization will severely penalize models unless Lao-aware tokenization (e.g., LaoNLP) is applied.
• Chain-of-thought (CoT) prompting improves performance on complex reasoning subdomains but yields minimal gains on factual or formulaic K12 questions, so evaluating only CoT or only direct-answer may skew results.
• Arena-style win-rates are sensitive to judge model family bias (e.g., Qwen3-Max favors Qwen-family models) and position bias; results must be averaged across judges and randomized positions to be reliable.

Evidence (verbatim from paper)

For K12 Education and Knowledge Application multiple-choice questions, we report Accuracy. For Translation tasks, we compute corpus-level BLEU against expert-written references under a standardized SacreBLEU configuration. Since Lao is written in a scriptio continua style without explicit word boundaries, BLEU can be sensitive to segmentation; we therefore apply Lao-aware tokenization using LaoNLP before scoring, and additionally report chrF++ in Appendix[C].

Citation

@misc{gao2025laobench,
  title={LaoBench: A Large-Scale Multidimensional Lao Benchmark for Large Language Models},
  author={Gao et al. (2025)},
  year={2025},
  note={arXiv:2511.11334}
}

• arXiv: 2511.11334