Bhasa Eval

bhasa-eval

BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Models — Leong et al. (2023) (arXiv:2309.06085, 2023)

What this evaluates

Evaluates large language models on Southeast Asian linguistic and cultural capabilities, probing syntax, semantics, pragmatics, coreference resolution, and scalar implicatures in Indonesian and Tamil.

Datasets

• BHASA LINDSEA (Indonesian & Tamil) — total ?; splits: test (-1); repo https://github.com/aisingapore/BHASA

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered instances out of the total. For binary choice tasks, it is calculated as (correct predictions / total instances) * 100. Runs are repeated three times; consistent correct/wrong answers are counted, while inconsistent ones are marked 'Unsure'.

Input / output format

Input: Linguistic diagnostic prompts including binary choice minimal pairs, blank-filling sentences, pronoun resolution questions, and scalar implicature/presupposition statements in Indonesian or Tamil.

Output: Model's selected option (A, B, True, or False) or generated text for open-ended tasks.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100
# Note: For multi-run tasks, only consistently correct/wrong answers are counted as correct/wrong; inconsistent runs are excluded or marked 'Unsure'.

Common pitfalls

• Models exhibit strong answer bias (e.g., preferring option B), which can inflate accuracy on binary choice tasks.
• Inconsistent answers across multiple runs are categorized as 'Unsure' rather than incorrect, potentially masking model instability.
• Translation artifacts in the XWinograd dataset may confound pragmatic and coreference evaluation.

Evidence (verbatim from paper)

Table 58: Breakdown of performance of GPT-3.5-Turbo and GPT-4 on the translated XWinograd binary choice task. The task was run thrice for each language. 'Correct' and 'Wrong' refer to answers that are consistently correct or wrong across all three runs. 'Unsure' refers to inconsistent answers across the three runs.

Citation

@misc{leong2023bhasa,
  title={BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Models},
  author={Leong et al. (2023)},
  year={2023},
  note={arXiv:2309.06085}
}

• arXiv: 2309.06085