Bengalimoralbench Eval

bengalimoralbench-eval

BengaliMoralBench: A Benchmark for Auditing Moral Reasoning in Large Language Models within Bengali Language and Culture — Ridoy et al. (2025) (arXiv:2511.03180, 2025)

What this evaluates

Evaluates large language models' ability to perform moral reasoning and align with human ethical judgments within Bengali language and South Asian socio-cultural contexts. It probes cultural grounding, commonsense reasoning, and fairness across five everyday moral domains using native-speaker consensus annotations.

Datasets

• BengaliMoralBench — total 3000; splits: test (3000); repo https://github.com/shahriyar-zaman/BengaliMoralBench.git

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted labels out of total instances.
• F1 score — range: percent
  • Harmonic mean of precision and recall, reported multiplied by 100.
• Matthews Correlation Coefficient (MCC) — range: [-1, 1]
  • Correlation coefficient between observed and predicted binary classifications, ranging from -1 to +1.
• Cohen’s Kappa — range: [-1, 1]
  • Statistical measure of inter-annotator agreement for qualitative items, adjusting for chance agreement.

Input / output format

Input: A moral scenario prompt in Bengali (or English) presented in a zero-shot format, asking the model to choose the ethically correct option or judgment.

Output: Model-generated text containing the selected classification label or reasoning leading to the final choice.

Scoring recipe

def compute_metrics(preds, golds):
    acc = sum(p == g for p, g in zip(preds, golds)) / len(golds)
    f1 = f1_score(golds, preds, average='weighted') * 100
    mcc = matthews_corrcoef(golds, preds)
    kappa = cohen_kappa_score(golds, preds)
    return {'accuracy': acc, 'F1 score': f1, 'MCC': mcc, 'Kappa': kappa}

Common pitfalls

• Using translated prompts instead of native Bengali introduces translation bias that artificially lowers performance.
• Evaluating only on Western-centric ethical frameworks ignores collectivist norms and cultural grounding specific to South Asian contexts.
• Reporting only accuracy without MCC or Kappa may mask class imbalance or chance-level agreement in multi-class moral judgments.

Evidence (verbatim from paper)

Model performance was measured using standard classification metrics: accuracy (%), F1 score (x100), Matthews Correlation Coefficient (MCC), and Cohen’s Kappa. These metrics provide a comprehensive assessment of the models’ ability to align with human ethical judgments across the benchmark.

Citation

@misc{ridoy2025bengalimoralbench,
  title={BengaliMoralBench: A Benchmark for Auditing Moral Reasoning in Large Language Models within Bengali Language and Culture},
  author={Ridoy et al. (2025)},
  year={2025},
  note={arXiv:2511.03180}
}

• arXiv: 2511.03180