Hindi LLM Benchmark Eval

hindi-llm-benchmark-eval

Benchmarking Hindi LLMs: A New Suite of Datasets and a Comparative Analysis — Kamath et al. (2025) (arXiv:2508.19831, 2025)

What this evaluates

Evaluates instruction-following, mathematical reasoning, code/function-calling, and retrieval-augmented generation capabilities of LLMs in Hindi. The benchmark specifically probes the models' ability to handle culturally and linguistically nuanced prompts that go beyond direct English translation.

Datasets

• IFEval-Hi — total ?; splits: (unstated)
• MT-Bench-Hi — total ?; splits: (unstated)
• GSM8K-Hi — total ?; splits: (unstated)
• ChatRAG-Hi — total ?; splits: (unstated)
• BFCL-Hi — total ?; splits: (unstated)

Metrics

• score (primary) — range: percent
  • Unspecified in the provided section; generally refers to task-specific pass rate or accuracy (e.g., exact match for math, instruction-following pass rate for IFEval).

Input / output format

Input: Hindi-language instruction prompts or queries, curated via a hybrid human-translation workflow to preserve linguistic and cultural nuance.

Output: Model-generated Hindi text responses.

Scoring recipe

for dataset in [IFEval-Hi, MT-Bench-Hi, GSM8K-Hi, ChatRAG-Hi, BFCL-Hi]:
  predictions = model.generate(prompts)
  metric_value = compute_task_specific_score(predictions, gold)
  record(metric_value)
aggregate_scores = mean(metric_values across datasets)
return aggregate_scores

Common pitfalls

• Activating dedicated reasoning modes (e.g., in GPT-OSS or Sarvam-M) can artificially inflate scores on complex tasks like BFCL-Hi and GSM8K-Hi if not strictly controlled.
• Using automated judges like GPT-4o may introduce bias toward sibling OpenAI models, skewing evaluation scores.
• Model size alone does not guarantee performance; targeted training and architectural choices heavily influence results across different task domains.

Evidence (verbatim from paper)

Gemma-2-9b-it provides the best all-around performance, securing the highest scores on MT-Bench-Hi, IFEval-Hi, and ChatRAG-Hi. ... the potential for the GPT-4o judge to be biased towards a sibling OpenAI model also warrants further investigation.

Citation

@misc{kamath2025benchmarkinghindi,
  title={Benchmarking Hindi LLMs: A New Suite of Datasets and a Comparative Analysis},
  author={Kamath et al. (2025)},
  year={2025},
  note={arXiv:2508.19831}
}

• arXiv: 2508.19831