Suvach Hindi QA Eval

suvach-hindi-qa-eval

Suvach -- Generated Hindi QA benchmark — Narayanan et al. (2024) (arXiv:2404.19254, 2024)

What this evaluates

Evaluates Hindi extractive question answering capabilities using multiple-choice questions generated from Wikipedia contexts. It probes a model's ability to comprehend Hindi text, locate relevant information, and select the correct answer from four options under varying context availability settings.

Datasets

• Suvach — total 100000; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard exact-match accuracy for multiple-choice questions. Calculated as the fraction of instances where the model's predicted choice exactly matches the gold answer.

Input / output format

Input: Hindi context (optional), Hindi question, and four multiple-choice options (optional). Evaluated in three settings: question only, question with context, and question with context and choices.

Output: A single selected choice from the four options.

Scoring recipe

correct = 0
total = 0
for instance in dataset:
    pred = model.generate(instance.context, instance.question, instance.choices)
    if pred == instance.answer:
        correct += 1
    total += 1
return correct / total

Common pitfalls

• The benchmark relies on LLM-generated questions validated by LLMs, which may not capture the full diversity or difficulty of human-annotated QA.
• Evaluation results vary significantly across the three input settings (with/without context), making cross-study comparisons difficult if the setting is not explicitly reported.

Evidence (verbatim from paper)

This dataset consists of over 100k question answers in Hindi, with 1200 tokens per question on average. The generated part of data contain Secret Context, Question, Choices, Answer, and Description. The question will be accompanied with 4 Choices and one and only one of them would be the correct answer. ... 1. Question only : Use only the question in evaluation prompt. 2. Question with context: Provide the context along with the question in evaluation prompt. 3. Question with context and choices: This would be the most easy task. The context and four possible answers are given in the evaluation prompt along with the question. Choose the most appropriate response.

Citation

@misc{narayanan2024suvach,
  title={Suvach -- Generated Hindi QA benchmark},
  author={Narayanan et al. (2024)},
  year={2024},
  note={arXiv:2404.19254}
}

• arXiv: 2404.19254