Zhuangbench Eval

zhuangbench-eval

Teaching Large Language Models an Unseen Language on the Fly — Chen Zhang et al. (2024) (arXiv:2402.19167, 2024)

What this evaluates

Evaluates large language models' ability to perform zero-shot machine translation into completely unseen, low-resource languages (Zhuang and Kalamang) using in-context learning. It probes how effectively models can adapt to new languages without prior training data by leveraging lexical expansion and syntactic exemplar retrieval.

Datasets

• ZhuangBench — total ?; splits: test (-1); repo https://github.com/luciusssss/ZhuangBench
• MTOB — total ?; splits: test (-1)

Metrics

• BLEU (primary) — range: percent
  • Standard n-gram precision with a brevity penalty, computed at the word level and scaled to 0-100.
• chrF — range: percent
  • Character-level F-score based on n-gram precision and recall, focusing on character n-grams.

Input / output format

Input: Source sentence in a known language (Chinese or English) paired with in-context learning exemplars retrieved via the DiPMT++ framework.

Output: Target sentence in the unseen language (Zhuang or Kalamang).

Scoring recipe

def compute_metrics(predictions, references):
    bleu = sacrebleu.corpus_bleu(predictions, [references]).score
    chrF = sacrebleu.corpus_chrf(predictions, [references]).score
    return {'BLEU': bleu, 'chrF': chrF}

Common pitfalls

• chrF can yield non-zero scores even for semantically meaningless outputs due to character overlap in low-resource Latin scripts.
• BLEU and chrF do not capture semantic correctness or fluency for completely unseen languages, potentially overestimating practical utility.
• Direct prompting without in-context exemplars yields near-zero scores, making the evaluation highly sensitive to the prompting protocol rather than just model capability.

Evidence (verbatim from paper)

Metrics. We use BLEU and chrF, implemented by Post (2018).. BLEU is a word-level metric while chrF focuses on the character level. ... In Table 1, we report the results on the Chinese-to-Zhuang (zh2za) and Zhuang-to-Chinese (za2zh) translation task of ZhuangBench. ... Although mT5-large achieves 15.1 chrF on zh2za, the model outputs are almost non-sense, as shown by the example in Appendix[D]. As Zhuang uses a Latin script with 26 characters, even a meaningless sequence would likely have a non-zero chrF score.

Citation

@misc{zhang2024teaching,
  title={Teaching Large Language Models an Unseen Language on the Fly},
  author={Chen Zhang et al. (2024)},
  year={2024},
  note={arXiv:2402.19167}
}

• arXiv: 2402.19167