# Bhasa Eval

> Evaluates large language models on Southeast Asian linguistic and cultural capabilities, probing syntax, semantics, pragmatics, coreference resolution, and scalar implicatures in Indonesian and Tamil. Use when the user wants to benchmark on BHASA LINDSEA (Indonesian & Tamil), or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/bhasa-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/bhasa-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/bhasa-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/bhasa-eval

---


# bhasa-eval

> BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Models — Leong et al. (2023) (arXiv:2309.06085, 2023)

## What this evaluates

Evaluates large language models on Southeast Asian linguistic and cultural capabilities, probing syntax, semantics, pragmatics, coreference resolution, and scalar implicatures in Indonesian and Tamil.

## Datasets

- **BHASA LINDSEA (Indonesian & Tamil)** — total ?; splits: test (-1); repo https://github.com/aisingapore/BHASA

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered instances out of the total. For binary choice tasks, it is calculated as (correct predictions / total instances) * 100. Runs are repeated three times; consistent correct/wrong answers are counted, while inconsistent ones are marked 'Unsure'.

## Input / output format

**Input**: Linguistic diagnostic prompts including binary choice minimal pairs, blank-filling sentences, pronoun resolution questions, and scalar implicature/presupposition statements in Indonesian or Tamil.

**Output**: Model's selected option (A, B, True, or False) or generated text for open-ended tasks.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100
# Note: For multi-run tasks, only consistently correct/wrong answers are counted as correct/wrong; inconsistent runs are excluded or marked 'Unsure'.
```

## Common pitfalls

- Models exhibit strong answer bias (e.g., preferring option B), which can inflate accuracy on binary choice tasks.
- Inconsistent answers across multiple runs are categorized as 'Unsure' rather than incorrect, potentially masking model instability.
- Translation artifacts in the XWinograd dataset may confound pragmatic and coreference evaluation.

## Evidence (verbatim from paper)

> Table 58: Breakdown of performance of GPT-3.5-Turbo and GPT-4 on the translated XWinograd binary choice task. The task was run thrice for each language. 'Correct' and 'Wrong' refer to answers that are consistently correct or wrong across all three runs. 'Unsure' refers to inconsistent answers across the three runs.

## Citation

```bibtex
@misc{leong2023bhasa,
  title={BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Models},
  author={Leong et al. (2023)},
  year={2023},
  note={arXiv:2309.06085}
}
```

- arXiv: 2309.06085