Crosslingual Mtf Eval

crosslingual-mtf-eval

Crosslingual Generalization through Multitask Finetuning — Muennighoff et al. (2022) (arXiv:2211.01786, 2022)

What this evaluates

Evaluates zero-shot crosslingual generalization of multilingual LLMs after multitask finetuning. Probes language-agnostic task understanding, robustness to prompt translation, and scaling behavior across NLU, generative, and code tasks.

Datasets

• XNLI — total ?; splits: test (-1)
• XCOPA — total ?; splits: test (-1)
• XStoryCloze — total ?; splits: test (-1)
• XWinograd — total ?; splits: test (-1)
• HumanEval — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted labels or completions for NLU tasks (XNLI, XCOPA, XStoryCloze, XWinograd, Sentence Completion, Coreference Resolution).
• BLEU — range: [0, 100]
  • Standard n-gram based metric for evaluating translation and summarization quality.
• pass@k — range: percent
  • Probability that at least one of k generated solutions passes the test suite. Evaluated at k=1, 10, 100 with 200 samples per problem across temperatures 0.2, 0.6, 0.8.

Input / output format

Input: Zero-shot prompted instances. Prompts are provided in English, machine-translated (MT), or human-translated (HT) versions of the target language. Tasks include NLI, causal reasoning, story cloze, coreference, translation, summarization, and code completion.

Output: Model generates a response or label corresponding to the prompt. For NLU/classification tasks, a discrete label. For generative tasks, a text completion. For HumanEval, a Python function completion.

Scoring recipe

def score(predictions, gold, task_type):
  if task_type == 'NLU':
    return sum(p == g for p, g in zip(predictions, gold)) / len(gold)
  elif task_type == 'NLG':
    return compute_bleu(gold, predictions)
  elif task_type == 'code':
    # Generate 200 samples per problem with top_p=0.95
    # Evaluate at temperatures 0.2, 0.6, 0.8
    # Return best pass@k across temperatures
    return pass_at_k(gold_tests, predictions, k)

Common pitfalls

• Using English prompts for non-English tasks without translation leads to significantly lower performance (e.g., BLOOMZ drops ~15% on XNLI MT prompts).
• Finetuning on short tasks induces a bias towards short generations, artificially lowering BLEU scores on generative tasks unless a minimum generation length is enforced at inference.
• Zero-shot performance on 'unseen' languages may still reflect minor pretraining contamination rather than true generalization.

Evidence (verbatim from paper)

The former are scored using accuracy and the latter using BLEU (Papineni et al., 2002). The NLG tasks measured are translation and summarization. ... Following Chen et al. (2021) we generate 200 samples for each problem with top p = 0.95 and compute pass rates. We perform this evaluation three times for temperatures 0.2, 0.6 and 0.8 and pick the best pass rate.

Citation

@misc{muennighoff2022crosslingual,
  title={Crosslingual Generalization through Multitask Finetuning},
  author={Muennighoff et al. (2022)},
  year={2022},
  note={arXiv:2211.01786}
}

• arXiv: 2211.01786