Indicmmlu Pro Eval

indicmmlu-pro-eval

IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding — Sankalp KJ et al. (2025) (arXiv:2501.15747, 2025)

What this evaluates

Evaluates large language models on multi-task language understanding across nine major Indic languages. It probes capabilities in reading comprehension, reasoning, and knowledge retention by adapting the English MMLU-Pro benchmark through machine translation and rigorous quality assurance.

Datasets

• IndicMMLU-Pro — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions out of the total number of items evaluated.
• chrF++ — range: percent
  • Character n-gram F1 score (n=6) measuring translation quality between original and back-translated text.
• BLEU — range: percent
  • N-gram precision score with a brevity penalty to penalize short translations.

Input / output format

Input: Multiple-choice question with four options (A, B, C, D) presented in one of nine Indic languages.

Output: A single selected option letter (A, B, C, or D) corresponding to the correct answer.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
    return (correct / len(gold_labels)) * 100

Common pitfalls

• The dataset was generated via machine translation (IndicTrans2) and back-translation, so translation artifacts or cultural mismatches may affect model performance independently of true language understanding.
• Quality assessment metrics (chrF++, BLEU, etc.) are only reported for three languages, leaving the remaining six languages without quantitative translation quality validation.
• Cosine similarity scores are used as a proxy for semantic alignment but do not replace direct human evaluation of translation fidelity.

Evidence (verbatim from paper)

Table 1: Performance comparison of language models on the IndicMMLU-Pro benchmark across nine Indic languages, including Indo-Aryan (Hindi, Bengali, Punjabi, Marathi, Urdu, and Gujarati) and Dravidian (Telugu, Tamil, and Kannada) languages. Accuracy scores are shown as percentages.

Citation

@misc{sankalp2025indicmmlupro,
  title={IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding},
  author={Sankalp KJ et al. (2025)},
  year={2025},
  note={arXiv:2501.15747}
}

• arXiv: 2501.15747