Indicxnli Eval

indicxnli-eval

IndicXNLI: Evaluating Multilingual Inference for Indian Languages — Aggarwal et al. (2022) (arXiv:2204.08776, 2022)

What this evaluates

Evaluates multilingual natural language inference capabilities across 11 Indic languages, probing both intra-lingual reasoning and cross-lingual transfer performance of pre-trained language models.

Datasets

• IndicXNLI — total ?; splits: train (-1), test (-1); repo https://github.com/divyanshuaggarwal/indicxnli

Metrics

• accuracy (primary) — range: percent
  • Standard classification accuracy: the proportion of correctly predicted inference labels (entailment, contradiction, neutral) out of the total number of instances.

Input / output format

Input: A pair of sentences: a premise and a hypothesis. Inputs are either monolingual in one of 11 Indic languages, or mixed-language (English premise with an Indic hypothesis).

Output: A single predicted label from the set {entailment, contradiction, neutral}.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
return (correct / len(gold_labels)) * 100

Common pitfalls

• The dataset is machine-translated from English XNLI, so evaluation scores conflate model reasoning ability with translation quality and semantic drift.
• Performance is highly sensitive to the fine-tuning strategy (e.g., English+Indic Train vs. Cross-lingual Transfer), so results are not directly comparable across different training protocols.
• Scores vary drastically based on language resource availability and script similarity, meaning low-resource languages (e.g., Odia, Assamese) will naturally underperform regardless of model architecture.

Evidence (verbatim from paper)

The classifiers take two sentence as input, i.e. the premise and the hypothesis and predicts the inference label. LangAvg represents the language wise average score across models, while ModAvg average score represents the model average score across languages.

Citation

@misc{aggarwal2022indicxnli,
  title={IndicXNLI: Evaluating Multilingual Inference for Indian Languages},
  author={Aggarwal et al. (2022)},
  year={2022},
  note={arXiv:2204.08776}
}

• arXiv: 2204.08776