Sailcompass Eval

sailcompass-eval

SailCompass: Towards Reproducible and Robust Evaluation for Southeast Asian Languages — Jia Guo et al. (2024) (arXiv:2412.01186, 2024)

What this evaluates

Evaluates large language models on language proficiency, reading comprehension, reasoning, and cultural understanding across three Southeast Asian languages (Indonesian, Vietnamese, Thai). It covers eight diverse tasks including question answering, machine translation, text summarization, multiple-choice exams, commonsense reasoning, machine reading comprehension, natural language inference, and sentiment analysis.

Datasets

• XQuAD — total 2318; splits: test (2318)
• TyDiQA — total 565; splits: val (565)
• Flores-200 — total 3036; splits: test (3036)
• ThaiSum — total 3671; splits: test (3671)
• IndoSum — total 3762; splits: test (3762)
• XLSUM — total 2676; splits: test (2676)
• M3Exam — total 375174; splits: test (375174)
• XCOPA — total 1500; splits: test (1500)
• BELEBELE — total 2700; splits: test (2700)
• XNLI — total 10020; splits: test (10020)
• IndoNLI — total 5182; splits: test (5182)
• Wisesight — total 2614; splits: test (2614)
• Indolem — total 1002; splits: test (1002)
• VSMEC — total 692; splits: test (692)

Metrics

• BLEU — range: [0, 100]
  • Standard n-gram overlap metric with a brevity penalty to penalize overly short translations. Scores are typically scaled to [0, 100].
• Chrf++ — range: [0, 100]
  • Character n-gram F-score that measures overlap at the character level, robust to morphological variations and tokenization differences.
• Exact Match (primary) — range: [0, 1]
  • Calculates the fraction of instances where the model's prediction exactly matches the gold label or answer span. Returns a value between 0 and 1.
• F1 Score — range: [0, 1]
  • Harmonic mean of precision and recall, calculated as 2 * (precision * recall) / (precision + recall). Used for classification tasks.

Input / output format

Input: Text, question, or passage in Indonesian, Vietnamese, or Thai, accompanied by task instructions and 1-3 few-shot examples translated into the target language.

Output: For generation tasks: a text sequence. For MCQ tasks: a selected option label. For classification tasks: a predicted category label.

Scoring recipe

def compute_metrics(predictions, golds, task_type):
    if task_type == 'generation':
        return {'bleu': bleu(golds, predictions), 'chrf': chrf(golds, predictions)}
    elif task_type in ['mcq', 'cls']:
        em = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
        tp = sum(1 for p, g in zip(predictions, golds) if p == g)
        fp = sum(1 for p in predictions if p not in golds)
        fn = sum(1 for g in golds if g not in predictions)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        return {'exact_match': em, 'f1': f1}

Common pitfalls

• Using instruction-tuned or chat models instead of base models, which contradicts the paper's goal of measuring pre-training upper bounds.
• Relying on translated English benchmarks rather than native-created datasets, which fails to properly assess cultural understanding and localized knowledge.
• Ignoring prompt calibration and few-shot example selection, which significantly impacts evaluation faithfulness for base models.

Evidence (verbatim from paper)

For Generation Tasks, we report BLEU*[[29]]* and Chrf++[[32]]. For MCQ Tasks, we report Exact Match. For Classification Tasks, we report Exact Match and F1 Score.

Citation

@misc{guo2024sailcompass,
  title={SailCompass: Towards Reproducible and Robust Evaluation for Southeast Asian Languages},
  author={Jia Guo et al. (2024)},
  year={2024},
  note={arXiv:2412.01186}
}

• arXiv: 2412.01186