Indicaleval Eval

indicaleval-eval

IndicEval: A Bilingual Indian Educational Evaluation Framework for Large Language Models — Bharti et al. (2026) (arXiv:2602.16467, 2026)

What this evaluates

Evaluates large language models' reasoning capabilities on authentic Indian high-stakes examination questions across STEM and humanities domains. It specifically probes bilingual reasoning, cross-lingual performance differentials, and the impact of prompting strategies (Zero-Shot, Few-Shot, Chain-of-Thought) on model accuracy.

Datasets

• IndicEval — total 1509; splits: test (1509)

Metrics

• exact-match accuracy (primary) — range: [0, 1]
  • Calculated as the proportion of model responses that exactly match the ground-truth correct option after deterministic parsing. Reported as the mean accuracy across three independent runs per model-prompt combination.

Input / output format

Input: Multiple-choice examination questions in either English or Hindi, presented via standardized prompt templates under Zero-Shot, Few-Shot, or Chain-of-Thought conditions.

Output: Model-generated text response containing the selected answer option, automatically parsed using a deterministic extraction module.

Scoring recipe

def compute_metric(predictions, gold):
    matches = sum(1 for p, g in zip(predictions, gold) if p == g)
    accuracy = matches / len(gold)
    return accuracy
# Final reported metric is the mean accuracy across 3 independent runs

Common pitfalls

• Failing to fix generation parameters (temperature, top-p, max tokens) across models, which introduces stochastic variance.
• Not preserving question order across models, potentially introducing ordering bias.
• Ignoring the need for deterministic answer parsing before exact-match scoring, leading to false negatives due to formatting differences.

Evidence (verbatim from paper)

Model responses were logged automatically, parsed using a deterministic extraction module, and evaluated using exact-match scoring. Mean accuracy across runs was reported as the primary performance metric, while variance was monitored to detect stochastic instability.

Citation

@misc{bharti2026indicaleval,
  title={IndicEval: A Bilingual Indian Educational Evaluation Framework for Large Language Models},
  author={Bharti et al. (2026)},
  year={2026},
  note={arXiv:2602.16467}
}

• arXiv: 2602.16467