Turkish Mt Eval

turkish-mt-eval

Mukayese: Turkish NLP Strikes Back — Safaya et al. (2022) (arXiv:2203.01215, 2022)

What this evaluates

Measures the quality of bidirectional translation between Turkish and English. It evaluates how well models capture cross-lingual semantic alignment and syntactic restructuring across different corpus types.

Datasets

• Wmt-16 (Turkish-English subset) — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/alisafaya/mukayese
• MuST-C (Turkish-English subset) — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/alisafaya/mukayese

Metrics

• BLEU Score (primary) — range: [0, 100]
  • Standard n-gram overlap metric (Papineni et al. 2002) computed over the test set. Evaluated in both tr-en and en-tr directions.

Input / output format

Input: Source language sentence (Turkish or English).

Output: Translated target language sentence.

Scoring recipe

def score_mt(predictions, gold):
    # Standard BLEU calculation
    bleu = compute_bleu(reference=[gold], hypothesis=predictions)
    return bleu * 100  # Paper reports scores on 0-100 scale

Common pitfalls

• BLEU scores are direction-dependent; tr-en and en-tr must be evaluated separately and reported distinctly.
• Pre-trained models like mBART50 are fine-tuned separately per dataset, so cross-dataset generalization is not measured.
• Results from scratch vs pre-trained models are reported in the same table but trained under different protocols.

Evidence (verbatim from paper)

We evaluate our models on the relevant test sets for translation in both directions. We utilize Bleu Score Papineni et al. (2002) for the assessment of translation quality.

Citation

@misc{safaya2022mukayese,
  title={Mukayese: Turkish NLP Strikes Back},
  author={Safaya et al. (2022)},
  year={2022},
  note={arXiv:2203.01215}
}

• arXiv: 2203.01215