qhjqhj00 igbo-english-mt-eval
Igbo-English Machine Translation: An Evaluation Benchmark — Ezeani et al. (2020) (arXiv:2004.00648, 2020)
What this evaluates
This benchmark evaluates bidirectional machine translation quality between Igbo and English. It probes a model's ability to accurately translate news and contemporary media content across two directions (Igbo-to-English and English-to-Igbo) using human-validated parallel sentences.
Datasets
- Igbo-English MT Benchmark — total 11584; splits: dev (10000), test (1000), hidden_test (584); repo https://github.com/IgnatiusEzeani/IGBONLP
Metrics
BLEU(primary) — range: [0, 100]- Standard n-gram overlap metric for machine translation. The paper does not explicitly define the metric in the provided section, but BLEU is the standard evaluation measure for MT benchmarks of this type.
Input / output format
Input: A single source sentence in either Igbo or English.
Output: A single target sentence in the corresponding target language (English or Igbo).
Scoring recipe
```python
import sacrebleu
def compute_bleu(predictions, references):
# predictions and references are lists of strings
score = sacrebleu.corpus_bleu(predictions, [references])
return score.score # Returns 0-100
## Common pitfalls
- The hidden test set (584 pairs) is not publicly released, preventing direct leaderboard comparison or full-dataset evaluation.
- The dataset is bidirectional (IG-EN and EN-IG); models must be evaluated separately for each direction to avoid cross-contamination.
- The ~380k monolingual Igbo sentences are provided for pre-training or unsupervised MT, not for direct supervised evaluation.
## Evidence (verbatim from paper)
> The splits of the parallel data into development, text and hidden test sets is shown in Table 2. Table 1: Breakdown of the Benchmark Evaluation Parallel Data ... Total 11,584
## Citation
```bibtex
@misc{ezeani2020igboenglish,
title={Igbo-English Machine Translation: An Evaluation Benchmark},
author={Ezeani et al. (2020)},
year={2020},
note={arXiv:2004.00648}
}
- arXiv: 2004.00648