Bangla Math Olympiad Eval

bangla-math-olympiad-eval

End-to-End Bangla AI for Solving Math Olympiad Problem Benchmark: Leveraging Large Language Model Using Integrated Approach — Tabib et al. (2025) (arXiv:2501.04425, 2025)

What this evaluates

Evaluates large language models' ability to solve mathematical Olympiad problems in Bangla and English. It probes multilingual reasoning, step-by-step problem solving, and the impact of retrieval-augmented generation and fine-tuning on low-resource language math tasks.

Datasets

• BDMO dataset — total 209; splits: test (209)
• Test dataset — total 100; splits: test (100)

Metrics

• accuracy (primary) — range: [0, 1] | percent
  • Number of correctly solved problems divided by the total number of problems in the evaluation set, reported as a fraction (e.g., X/Y) or percentage.

Input / output format

Input: Math Olympiad problems presented in either Bangla or English, optionally accompanied by system prompts or retrieval-augmented context.

Output: Step-by-step reasoning trace and final answer/solution to the math problem.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if pred.strip().lower() == gold.strip().lower():
            correct += 1
    return correct / len(gold_answers)

Common pitfalls

• The paper reports raw counts (e.g., 115/209) rather than normalized percentages, which can be confusing when comparing across different dataset sizes.
• Language pairing (problem language vs. reasoning language) significantly impacts scores, but the paper does not standardize a single evaluation condition, making cross-model comparisons difficult without careful configuration matching.

Evidence (verbatim from paper)

GPT4o-TIR had the highest accuracy of 130 out of 209 in BDMO dataset. When Bangla was the main language, the model performed worse, suggesting the need for more advanced and versatile model.

Citation

@misc{tabib2025endtoendbangla,
  title={End-to-End Bangla AI for Solving Math Olympiad Problem Benchmark: Leveraging Large Language Model Using Integrated Approach},
  author={Tabib et al. (2025)},
  year={2025},
  note={arXiv:2501.04425}
}

• arXiv: 2501.04425