Dzen Eval

dzen-eval

Multilingual Question Answering in Low-Resource Settings: A Dzongkha-English Benchmark for Foundation Models — Hosain et al. (2025) (arXiv:2505.18638, 2025)

What this evaluates

Evaluates foundation models' ability to answer multiple-choice academic questions in both English and Dzongkha across varying grade levels and scientific subjects. It specifically probes factual recall, procedural application, and multi-step reasoning capabilities in a low-resource multilingual setting.

Datasets

• DZEN — total 5161; splits: test (5161); repo https://github.com/kraritt/llm_dzongkha_evaluation

Metrics

• accuracy (primary) — range: [0, 1]
  • Exact match accuracy: the proportion of questions where the model's predicted option exactly matches the ground truth answer. Calculated as correct predictions divided by total questions.

Input / output format

Input: Multiple-choice question presented in either Dzongkha or English, containing four possible answer options. Mathematical and chemical expressions are formatted in LaTeX.

Output: Selection of one of the four provided options (e.g., A, B, C, or D) corresponding to the correct answer.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_answers):
    if pred.strip().upper() == gold.strip().upper():
        correct += 1
accuracy = correct / len(gold_answers)

Common pitfalls

• Question ordering differs between English and Dzongkha versions, requiring semantic matching rather than positional alignment.
• Ground truth discrepancies may arise from translation errors or annotator mistakes, necessitating manual verification of parallel pairs.
• Grammar mistakes in English translations can occasionally affect model performance, though the paper notes the effect is often minimal.

Evidence (verbatim from paper)

Chain-of-Thought prompting improves reasoning accuracy in Dzongkha when applied directly in the target language, especially when paired with English translations for factual and multi-step questions, while translation-augmented prompts enhance response precision without sacrificing linguistic accessibility.

Citation

@misc{hosain2025dzen,
  title={Multilingual Question Answering in Low-Resource Settings: A Dzongkha-English Benchmark for Foundation Models},
  author={Hosain et al. (2025)},
  year={2025},
  note={arXiv:2505.18638}
}

• arXiv: 2505.18638