Cendol Eval

cendol-eval

Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languages — Cahyawijaya et al. (2024) (arXiv:2404.06138, 2024)

What this evaluates

This evaluation protocol assesses the language proficiency, generalization capability, and local cultural commonsense reasoning of instruction-tuned LLMs across Indonesian and nine indigenous languages. It probes zero-shot performance on seen and unseen tasks and languages, as well as nuanced understanding of regional proverbs, figures of speech, and story endings.

Datasets

• COPAL-ID — total ?; splits: test (-1)
• MABL — total ?; splits: test (-1)
• IndoStoryCloze — total ?; splits: test (-1)
• MAPS — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Task-dependent; calculated as the proportion of correctly predicted options or generated completions against gold labels in zero-shot settings.

Input / output format

Input: Zero-shot prompts containing scenarios, questions, or sentence fragments with multiple-choice options or open-ended completion targets in Indonesian or indigenous languages.

Output: Model-generated text selecting the most plausible option, completing the story, or interpreting the figure of speech/proverb.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if normalize(pred) == normalize(gold):
        correct += 1
return correct / len(gold_labels)

Common pitfalls

• Zero-shot evaluation may not reflect performance after instruction tuning or fine-tuning.
• Cultural commonsense datasets require deep regional knowledge that generic multilingual models often lack, leading to systematic failures on local nuances.
• Code-switching and dialectal variations in indigenous language datasets can cause tokenization mismatches and degraded performance.

Evidence (verbatim from paper)

To assess the language capability of Cendol models across Indonesian indigenous languages, we design an evaluation benchmark with 15 datasets covering 10 languages including Indonesian and 9 local languages spoken in Indonesia, i.e., Acehnese (ace), Balinese (ban), Banjarese (bjn), Buginese (bug), Javanese (jav), Madurese (mad), Minangkabau (min), Ngaju (nij), and Sundanese (sun). For all tasks and datasets, we evaluate the model in a zero-shot prompting setting.

Citation

@misc{cahyawijaya2024cendol,
  title={Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languages},
  author={Cahyawijaya et al. (2024)},
  year={2024},
  note={arXiv:2404.06138}
}

• arXiv: 2404.06138