Xnli Sib200 Multilingual Eval

xnli-sib200-multilingual-eval

Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settings — Deya et al. (2024) (arXiv:2410.13153, 2024)

What this evaluates

Evaluates multilingual and cross-lingual capabilities of LLMs on Natural Language Inference (XNLI) and topic classification (SIB-200) across English and low-resource South Asian languages (Bangla, Hindi, Urdu) using zero-shot prompting.

Datasets

• XNLI — total ?; splits: test (-1)
• SIB-200 — total ?; splits: test (-1)

Metrics

• Accuracy — range: percent
  • Proportion of correctly predicted class labels out of the total number of samples.
• F1macro (primary) — range: percent
  • Macro-averaged F1 score computed across all prediction classes, treating each class equally regardless of support size.

Input / output format

Input: Zero-shot prompts in English, Bangla, Hindi, or Urdu, including original and cross-lingually translated variants (settings P1–P5).

Output: Predicted class label (e.g., 'Entailment', 'Contradiction', 'Neutral' for XNLI; topic names for SIB-200), often accompanied by extraneous text or invalid characters requiring post-processing.

Scoring recipe

def evaluate(predictions, gold):
    clean_preds = [extract_valid_label(p) for p in predictions]
    # Map invalid/unknown labels to inverse class as per paper protocol
    clean_preds = [map_invalid(p) if is_invalid(p) else p for p in clean_preds]
    acc = sum(1 for p, g in zip(clean_preds, gold) if p == g) / len(gold)
    f1 = macro_f1(gold, clean_preds)
    return {'accuracy': acc * 100, 'f1macro': f1 * 100}

Common pitfalls

• LLMs frequently return extraneous characters or invalid labels alongside class predictions, requiring strict post-processing before scoring.
• Safety filters (e.g., in Gemini Pro) may block harmful content samples, resulting in missing predictions that must be tracked and handled.
• Cross-lingual prompt translation does not reliably improve performance; English prompts consistently outperform translated low-resource variants.

Evidence (verbatim from paper)

Table 4: Class-wise F1macro score for GPT-4, Llama 2, and Gemini across five prompt settings for the XNLI dataset. Lang.: Language, BN: Bangla, EN: English, HI: Hindi, and UR: Urdu, Cont: contradiction, Ent: Entailment, Neut: Neutral.

Citation

@misc{deya2024better,
  title={Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settings},
  author={Deya et al. (2024)},
  year={2024},
  note={arXiv:2410.13153}
}

• arXiv: 2410.13153