Indic Instruct Eval

indic-instruct-eval

UPDESH: Synthesizing Grounded Instruction Tuning Data for 13 Indic — Chitale et al. (2025) (arXiv:2509.21294, 2025)

What this evaluates

This evaluation probes the multilingual instruction-following, natural language understanding, and generation capabilities of LLMs fine-tuned on 13 Indic languages. It measures performance on standardized academic benchmarks across NLU and NLG tasks, as well as real-world cultural relevance and helpfulness through pairwise LLM-as-a-judge comparisons.

Datasets

• MMLU Indic (MMLU-I) — total ?; splits: test (-1)
• ARC Indic (ARC-I) — total ?; splits: test (-1)
• BoolQ Indic (BoolQ-I) — total ?; splits: test (-1)
• TriviaQA Indic (TVQA-I) — total ?; splits: test (-1)
• BeleBele (Bele) — total ?; splits: test (-1)
• INCLUDE (INCL) — total ?; splits: test (-1)
• Global MMLU (GMMLU) — total ?; splits: test (-1)
• Extreme Summarization (Xsum) — total ?; splits: test (-1)
• Flores EnXX / XXEn — total ?; splits: test (-1)
• IN22-Conv-Doc — total ?; splits: test (-1)

Metrics

• NLU average score — range: [0, 1]
  • Likelihood-based scoring on multiple-choice questions; accuracy or log-probability selection of the correct option averaged across all NLU benchmarks.
• NLG average score — range: [0, 1]
  • Average of standard generation metrics (e.g., ROUGE/BERTScore) across translation and summarization tasks.
• ELO rating (primary) — range: other
  • Pairwise LLM-as-a-judge comparison score updated iteratively; higher ratings indicate better comparative performance across real-world, culturally grounded queries.

Input / output format

Input: Instruction prompts in 13 Indic languages (and English for cross-lingual tasks), formatted as multiple-choice questions for NLU or translation/summarization prompts for NLG.

Output: Model-generated text responses (selected option for NLU, translated/summarized text for NLG).

Scoring recipe

def score_nlu(dataset, model):
    correct = 0
    for q in dataset:
        options = q.options
        best_opt = max(options, key=lambda o: model.log_prob(q.prompt + o))
        if best_opt == q.gold:
            correct += 1
    return correct / len(dataset)

def score_nlg(dataset, model):
    scores = [compute_generation_metric(q.gold, model.generate(q.prompt)) for q in dataset]
    return sum(scores) / len(scores)

def update_elo(models, battles, llm_judge):
    for a, b, prompt in battles:
        judge_win = llm_judge(prompt, models[a].generate(prompt), models[b].generate(prompt))
        elo_update(a, b, judge_win)
    return {m: elo_rating(m) for m in models}

Common pitfalls

• Failing to subsample large baseline datasets (e.g., Aya-Collection, IndicAlign) to match the target dataset size, creating an unfair data/compute advantage.
• Overlooking cross-lingual transfer evaluation on languages completely absent from the training set (e.g., the 16 unseen languages in the Flores subset).
• Relying exclusively on automated academic benchmarks without incorporating LLM-as-a-judge cultural relevance evaluations, which miss real-world helpfulness.

Evidence (verbatim from paper)

Natural language understanding (NLU) tasks use multiple-choice questions to measure comprehension and reasoning through likelihood-based scoring. Natural language generation (NLG) tasks, such as translation and summarization, assess models’ ability to generate coherent and contextually appropriate outputs. We augment standard dataset-NLU and NLG evaluations with comparative evaluations to understand model win rates.

Citation

@misc{chitale2025updesh,
  title={UPDESH: Synthesizing Grounded Instruction Tuning Data for 13 Indic},
  author={Chitale et al. (2025)},
  year={2025},
  note={arXiv:2509.21294}
}

• arXiv: 2509.21294