qhjqhj00 byol-lrl-eval
BYOL: Bring Your Own Language Into LLMs — Zamir et al. (2026) (arXiv:2601.10804, 2026)
What this evaluates
Evaluates LLM capabilities in low- and extreme-low-resource languages (Chichewa, Māori) across reasoning, reading comprehension, factual knowledge, and machine translation. It measures both language-specific adaptation gains and preservation of multilingual/English capabilities.
Datasets
- BYOL Evaluation Benchmarks — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Proportion of correct predictions out of total instances. Calculated as correct / total.
average_score— range: percent- Mean of all benchmark metrics normalized to the [0,1] range before averaging.
BLEU— range: percent- Standard n-gram overlap metric for machine translation evaluation.
chrF++— range: percent- Character n-gram F-score used for translation evaluation.
win-rate— range: percent- Percentage of pairwise comparisons where the model's response is preferred by the LLM-as-a-judge under a forced-choice protocol.
Input / output format
Input: Text prompts or multiple-choice questions in Chichewa, Māori, or English, formatted for the lm-evaluation-harness framework.
Output: Model-generated text responses or selected multiple-choice options, evaluated against gold reference answers.
Scoring recipe
def compute_metrics(predictions, golds, metric_type):
if metric_type == 'accuracy':
return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
elif metric_type in ['BLEU', 'chrF++']:
return compute_mt_metric(predictions, golds)
elif metric_type == 'win-rate':
return sum(1 for p in predictions if p == 'winner') / len(predictions)
return 0.0
def compute_average_score(benchmark_results):
normalized = [min(max(r, 0.0), 1.0) for r in benchmark_results.values()]
return sum(normalized) / len(normalized)
Common pitfalls
- Normalizing disparate metrics (accuracy, BLEU, chrF++) to [0,1] before averaging can obscure task-specific performance differences.
- Machine-translated benchmarks may introduce translation artifacts that confound model capability assessment.
- LLM-as-a-judge forced-choice protocol eliminates ties, potentially inflating win-rates for marginally better models.
Evidence (verbatim from paper)
Throughout the paper, we report scores for each benchmark/task using its standard evaluation metric (accuracy, BLEU, chrF++, etc.). The average score (reported as a percentage) is computed by normalizing each metric to the $[0,1]$ range and using chrF++ for the translation task. All translated benchmarks are integrated into the lm-evaluation-harness framework.
Citation
@misc{zamir2026byol,
title={BYOL: Bring Your Own Language Into LLMs},
author={Zamir et al. (2026)},
year={2026},
note={arXiv:2601.10804}
}
- arXiv: 2601.10804