Titullms Bangla Benchmark Eval

titullms-bangla-benchmark-eval

TituLLMs: A Family of Bangla LLMs with Comprehensive Benchmarking — Shahriar Kabir Nahin et al. (2025) (arXiv:2502.11187, 2025)

What this evaluates

Evaluates large language models on Bangla language capabilities, specifically probing world knowledge, commonsense reasoning, physical reasoning, and reading comprehension. The benchmark uses multiple-choice and yes/no question formats to measure how well models understand and generate text in a low-resource language context.

Datasets

• Bangla MMLU — total 87869; splits: val (72944), test (14750), dev (175); repo https://github.com/hishab-nlp/titulm
• CommonsenseQA BN — total 10962; splits: train (9741), val (1221); repo https://github.com/hishab-nlp/titulm
• OpenBookQA BN — total 5944; splits: train (4947), val (500), test (497); repo https://github.com/hishab-nlp/titulm
• PIQA BN — total 17177; splits: train (15339), val (1838); repo https://github.com/hishab-nlp/titulm
• BoolQ BN — total 1976; splits: train (815), val (432), test (729); repo https://github.com/hishab-nlp/titulm

Metrics

• normalized accuracy (primary) — range: [0, 1]
  • Overall accuracy calculated as the total number of correct predictions divided by the total number of examples across all benchmark subsets. Predictions are matched exactly against the gold labels.

Input / output format

Input: Multiple-choice questions with four options, or yes/no reading comprehension questions paired with a context passage. Prompts are formatted according to lm-evaluation-harness task definitions.

Output: Model generates a text completion or selects the correct option label. Evaluated via lm-evaluation-harness using normalized accuracy.

Scoring recipe

correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().lower() == gold.strip().lower():
        correct += 1
    total += 1
return correct / total if total > 0 else 0.0

Common pitfalls

• Several subsets (CommonsenseQA BN, PIQA BN) lack a dedicated test split, containing only train and validation data, which may lead to overestimation if validation data is used for tuning or selection.
• The paper reports a single aggregate normalized accuracy across all subsets, which masks per-dataset performance variations and may hide weaknesses in specific reasoning categories.
• Datasets rely on machine translation (EST method) or GPT-4 generation, so translation artifacts or synthetic noise may artificially inflate or deflate model scores compared to native human-annotated benchmarks.

Evidence (verbatim from paper)

For evaluation, we utilized the lm-evaluation-harness. We used normalized accuracy as a metric. Our assessment focuses on key aspects such as knowledge and reasoning. We benchmarked TituLLMs alongside other popular LLMs using five newly prepared evaluation datasets. The dataset is composed of multiple subsets of the benchmarking set, including Bangla MMLU (87,869 entries), Piqa BN (17,177 entries), CommonsenseQA BN (10,962 entries), OpenBookQA BN (5,944 entries), and BoolQ BN (1,976 entries).

Citation

@misc{nahin2025titullms,
  title={TituLLMs: A Family of Bangla LLMs with Comprehensive Benchmarking},
  author={Shahriar Kabir Nahin et al. (2025)},
  year={2025},
  note={arXiv:2502.11187}
}

• arXiv: 2502.11187