Chikha Po Eval

chikha-po-eval

ChiKhaPo: A Large-Scale Multilingual Benchmark for Evaluating Lexical Comprehension and Generation in Large Language Models — Chang et al. (2025) (arXiv:2510.16928, 2025)

What this evaluates

Evaluates fundamental lexical comprehension and generation capabilities of multilingual LLMs across thousands of languages. It probes word-level translation, context-aware translation, translation-conditioned language modeling, and bag-of-words machine translation tasks to measure basic linguistic competence beyond high-resource languages.

Datasets

• ChiKhaPo — total ?; splits: test (-1)

Metrics

• language score (primary) — range: [0, 1]
  • Accuracy computed over a minimum of 100 words per language. Scores are aggregated across languages after applying dataset caps.

Input / output format

Input: Word-level prompts for translation tasks (WT, WTWC) and translation-conditioned language modeling/bag-of-words MT tasks.

Output: Predicted target word or sequence per prompt.

Scoring recipe

scores = {}
for lang in languages:
    if task in [WT, WTWC]: limit to 300 words
    if task in [TCLM, BOW MT]: use 30% of data
    if len(gold) >= 100:
        scores[lang] = accuracy(predictions, gold)

Common pitfalls

• Evaluation uses capped subsets (300 words/language for WT/WTWC, 30% for TCLM/BOW MT) rather than full datasets.
• Reported scores require a minimum of 100 words per language to be valid.
• Performance is strongly influenced by language resourcedness, family, and task type.

Evidence (verbatim from paper)

All reported language scores across are computed over a minimum of 100 words per language.

Citation

@misc{chang2025chikha-po,
  title={ChiKhaPo: A Large-Scale Multilingual Benchmark for Evaluating Lexical Comprehension and Generation in Large Language Models},
  author={Chang et al. (2025)},
  year={2025},
  note={arXiv:2510.16928}
}

• arXiv: 2510.16928