Ebible Benchmarks Eval

ebible-benchmarks-eval

The eBible Corpus: Data and Model Benchmarks for Bible Translation for Low-Resource Languages — Åkerman et al. (2023) (arXiv:2304.09919, 2023)

What this evaluates

Machine translation performance on low-resource languages using verse-aligned Bible texts. It probes model robustness across different biblical book genres (Gospels, Epistles, OT books) and the utility of related language data for translation.

Datasets

• eBible Corpus — total ?; splits: train (-1), test (-1); repo https://github.com/BibleNLP/ebible

Metrics

• BLEU (primary) — range: percent
  • Standard n-gram precision metric with brevity penalty. Calculated at the sentence/verse level.
• spBLEU — range: percent
  • Sentence-piece BLEU, which computes BLEU over subword tokenizations to handle morphologically rich or low-resource languages better.
• chrF3 — range: percent
  • Character n-gram F-score with n=3, measuring character-level precision and recall.

Input / output format

Input: Source language Bible verse text.

Output: Target language translated Bible verse text.

Scoring recipe

def compute_metrics(predictions, references):
    bleu = sacrebleu.corpus_bleu(predictions, [references]).score
    sp_bleu = sacrebleu.corpus_bleu(predictions, [references], tokenize='spm').score
    chrf3 = sacrebleu.corpus_chrf(predictions, [references], char_order=3).score
    return {'BLEU': bleu, 'spBLEU': sp_bleu, 'chrF3': chrf3}

Common pitfalls

• BLEU scores can vary significantly from subword metrics (spBLEU) for morphologically complex languages, so relying solely on BLEU may misrepresent performance.
• The benchmark uses verse-aligned data, so evaluation must be done at the verse level, not paragraph or chapter level.
• Performance varies widely across biblical book genres (Gospels vs. Epistles vs. OT), so reporting a single aggregate score without genre breakdown is misleading.

Evidence (verbatim from paper)

Figure 6 shows a bar chart of the median BLEU, spBLEU, and chrF3 scores for all eight translation pairings. Interestingly, we find no clear correlation between the scope (NT-only, NT with partial OT, or full Bible) of the translation pairing and our selected scoring metrics.

Citation

@misc{akerman2023ebible,
  title={The eBible Corpus: Data and Model Benchmarks for Bible Translation for Low-Resource Languages},
  author={Åkerman et al. (2023)},
  year={2023},
  note={arXiv:2304.09919}
}

• arXiv: 2304.09919