Bayling2 Multilingual Eval

bayling2-multilingual-eval

BayLing 2: A Multilingual Large Language Model with Efficient Language Alignment — Zhang et al. (2024) (arXiv:2411.16300, 2024)

What this evaluates

Evaluates multilingual translation quality and cross-lingual reasoning across high-resource and low-resource languages. It probes the model's ability to align languages and transfer capabilities from high-resource to low-resource settings without extensive low-resource instruction data.

Datasets

• Flores-101 — total ?; splits: test (-1)
• WMT22 — total ?; splits: test (-1)
• Belebele — total ?; splits: test (-1)
• XNLI — total ?; splits: test (-1)
• GSM8K — total ?; splits: test (-1)

Metrics

• BLEU (sacrebleu) (primary) — range: percent
  • Measures statistical similarity based on n-gram precision between the generated translation and reference translations, typically reported as a percentage.
• COMET (primary) — range: percent
  • Measures semantic similarity using cross-lingual pre-trained models, regarded as the most human-aligned evaluation metric for translation tasks.
• Accuracy — range: [0, 1]
  • Calculates the proportion of correctly selected options in multiple-choice benchmarks.

Input / output format

Input: Source language sentence (for translation) or multiple-choice question with options (for multi-task/general benchmarks).

Output: Target language sentence (for translation) or selected option letter/text (for multi-task/general benchmarks).

Scoring recipe

def score(predictions, golds, metric):
    if metric == 'bleu':
        return sacrebleu.corpus_bleu(predictions, [golds]).score
    elif metric == 'comet':
        return comet_model.predict(predictions, golds).score
    else:  # accuracy
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds)

Common pitfalls

• Flores-101 low-resource languages require a 1-shot in-context example to prevent off-target generation, unlike the 0-shot setting used for other benchmarks.
• COMET evaluates semantic similarity via cross-lingual models rather than surface n-gram overlap, so it can diverge significantly from BLEU scores.

Evidence (verbatim from paper)

For metrics, BLEU (sacrebleu) (Post, [2018]) and COMET (Rei et al., [2022]) are used to assess the quality of LLMs’ translation. BLEU score measures the statistical similarity based on n-gram accuracy, COMET score measures the semantic similarity using cross-lingual pre-trained models, which is currently regarded as the most human-aligned evaluation metric for translation tasks.

Citation

@misc{zhang2024bayling2,
  title={BayLing 2: A Multilingual Large Language Model with Efficient Language Alignment},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2411.16300}
}

• arXiv: 2411.16300