Bianet Mt Eval

bianet-mt-eval

Bianet: A Parallel News Corpus in Turkish, Kurdish and English — Ataman et al. (2018) (arXiv:1805.05095, 2018)

What this evaluates

Evaluates the impact of the Bianet parallel corpus on Neural Machine Translation performance for English-Turkish and English-Kurdish language pairs in the news domain. It compares baseline models trained on existing corpora against models augmented with Bianet data, and assesses multilingual transfer learning benefits.

Datasets

• WMT2016 — total 4001; splits: dev (1001), test (3000)
• Bianet — total 48956; splits: train (48956)
• SETIMES — total 205706; splits: train (205706)
• Ubuntu & GNUME — total 65357; splits: train (65357)

Metrics

• BLEU (primary) — range: [0, 100]
  • Standard n-gram precision with brevity penalty, computed corpus-wide. Reported on a 0-100 scale.
• chrF3 — range: [0, 1]
  • Character n-gram F-score with n=3, computed corpus-wide.

Input / output format

Input: Source sentence in English, Turkish, or Kurdish.

Output: Target sentence in the corresponding target language (Turkish, English, or Kurdish).

Scoring recipe

def evaluate(predictions, references):
    # predictions and references are lists of strings
    bleu = corpus_bleu(predictions, [references]).score
    chrf = corpus_chrf(predictions, [references], order=3).score
    return {'BLEU': bleu, 'chrF3': chrf}

Common pitfalls

• The English-Kurdish evaluation uses a custom sampled split from Bianet (500 dev, 500 test) because no official WMT test set exists for this pair, unlike English-Turkish which uses WMT2016.
• BLEU scores are reported on a 0-100 scale in the tables, not the 0-1 scale used in some implementations.
• Models are trained with subword segmentation (BPE) and evaluated on detokenized output; mismatched tokenization can skew chrF3/bleu.

Evidence (verbatim from paper)

We use the BLEU (Papineni et al., 2002) and chrF3 (Popovic, 2015) automatic evaluation metrics and the Multeval (Clark et al., 2011) significance test for evaluating the accuracy of the models. The translation accuracy obtained on the WMT Turkish testing sets are given in Table 4.

Citation

@misc{ataman2018bianet,
  title={Bianet: A Parallel News Corpus in Turkish, Kurdish and English},
  author={Ataman et al. (2018)},
  year={2018},
  note={arXiv:1805.05095}
}

• arXiv: 1805.05095