Tlue Eval

tlue-eval

TLUE: A Tibetan Language Understanding Evaluation Benchmark — Gao et al. (2025) (arXiv:2503.12051, 2025)

What this evaluates

Evaluates large language models' proficiency in Tibetan across general knowledge comprehension and safety-critical domains. It probes the models' ability to handle low-resource language tasks, complex reasoning, and culturally sensitive alignment compared to English baselines.

Datasets

• Ti-MMLU — total ?; splits: test (-1); repo https://github.com/Vicentvankor/TLUE
• Ti-SafetyBench — total ?; splits: test (-1); repo https://github.com/Vicentvankor/TLUE

Metrics

• Accuracy (ACC) (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions out of the total dataset size.
• Conditional Accuracy (CA) — range: percent
  • Percentage of correctly answered questions among only those instances where the model provides a direct answer (excluding refusals or non-responses).
• Response Rate (RR) — range: percent
  • Percentage of instances where the model produces a non-refusal answer out of the total dataset size.

Input / output format

Input: Multiple-choice questions in Tibetan covering 67 academic/professional subjects (Ti-MMLU) or safety-critical scenarios (Ti-SafetyBench).

Output: Model-generated text response, typically a selected option or a direct answer to the prompt.

Scoring recipe

def calculate_metrics(predictions, golds, refusals):
    total = len(golds)
    rr = sum(1 for p in predictions if p not in refusals) / total
    answered = [(p, g) for p, g in zip(predictions, golds) if p not in refusals]
    acc = sum(1 for p, g in answered if p == g) / total
    ca = sum(1 for p, g in answered if p == g) / len(answered) if answered else 0
    return rr, acc, ca

Common pitfalls

• Models often refuse to answer safety-related prompts in Tibetan, drastically lowering raw Accuracy; Conditional Accuracy (CA) must be used to fairly assess knowledge when refusals occur.
• The benchmark adapts Chinese benchmarks (CMMLU, SafetyBench) via translation, so performance gaps may stem from translation quality or cultural mismatch rather than pure language proficiency.
• Random baselines differ between tasks (25% for Ti-MMLU, ~36.7% for Ti-SafetyBench), so comparing raw scores across domains without accounting for option counts or refusal rates is misleading.

Evidence (verbatim from paper)

Table 4: List of Abbreviations for Professional Terms | Response Rate | RR | | Accuracy | ACC | | Conditional Accuracy | CA |

Citation

@misc{gao2025tlue,
  title={TLUE: A Tibetan Language Understanding Evaluation Benchmark},
  author={Gao et al. (2025)},
  year={2025},
  note={arXiv:2503.12051}
}

• arXiv: 2503.12051