Kokborok Mt Eval

kokborok-mt-eval

Towards High-Quality Machine Translation for Kokborok: A Low-Resource Tibeto-Burman Language of Northeast India — Nyalang et al. (2026) (arXiv:2604.19778, 2026)

What this evaluates

Evaluates machine translation quality for Kokborok (a low-resource Tibeto-Burman language) in both English-to-Kokborok and Kokborok-to-English directions. It probes translation adequacy, fluency, and semantic similarity using both automatic metrics and human ratings.

Datasets

• SMOL Test Set — total ?; splits: test (-1)
• WMT Test Set (Bible domain) — total ?; splits: test (-1)

Metrics

• BLEU (primary) — range: percent
  • Corpus-level Bilingual Evaluation Understudy score computed via sacreBLEU with default tokenization. Measures n-gram overlap between system output and reference translations.
• chrF — range: percent
  • Character n-gram F-score computed via sacreBLEU. Measures character-level n-gram overlap, robust to morphological variations.
• ROUGE-L — range: percent
  • Longest common subsequence F-measure. Measures the longest chain of matching words in order between prediction and reference.
• METEOR — range: percent
  • Alignment-based metric using WordNet for synonym matching and stemming. Designed to correlate better with human judgment than BLEU.
• TER — range: other
  • Translation Edit Rate. Measures the number of edits (insertions, deletions, substitutions, shifts) needed to change the hypothesis into the reference, divided by reference length.
• Cosine Similarity — range: [0, 1]
  • Semantic similarity computed using LaBSE (Language-agnostic BERT Sentence Embeddings) to measure vector space proximity between source and target sentences.
• COMET — range: [0, 1]
  • Neural evaluation metric using the Unbabel/wmt22-comet-da model. Predicts translation quality scores based on contextual embeddings of source, hypothesis, and reference.
• Adequacy — range: 1-5
  • Human-rated scale (1-5) measuring whether the translation preserves the meaning of the source sentence.
• Fluency — range: 1-5
  • Human-rated scale (1-5) measuring whether the translation is natural and grammatically correct in the target language.

Input / output format

Input: Source sentence in English or Kokborok (trp)

Output: Translated sentence in Kokborok (trp) or English

Scoring recipe

def score_mt(predictions, references):
    # Automatic metrics via sacreBLEU
    bleu = sacrebleu.corpus_bleu(predictions, [references]).score
    chrf = sacrebleu.corpus_chrf(predictions, [references]).score
    rouge_l = sacrebleu.corpus_rouge(predictions, [references]).score.fmeasure
    meteor = sacrebleu.corpus_meteor(predictions, [references]).score
    ter = sacrebleu.corpus_ter(predictions, [references]).score  # lower is better
    # Semantic & Neural
    cosine_sim = cosine_similarity(labse_embed(predictions), labse_embed(references))
    comet_score = comet_model.predict(predictions, references)
    return {'BLEU': bleu, 'chrF': chrf, 'ROUGE-L': rouge_l, 'METEOR': meteor, 'TER': ter, 'Cosine': cosine_sim, 'COMET': comet_score}

Common pitfalls

• TER is lower-is-better while all other automatic metrics are higher-is-better, which can cause sign errors in aggregation.
• LaBSE-based Cosine Similarity fails for Kokborok due to the language's absence in LaBSE's training data, making semantic similarity scores unreliable.
• Human evaluation used a small sample (n=50) with mixed expertise, leading to low inter-annotator agreement for the untrained native speaker (κ=0.13).

Evidence (verbatim from paper)

We evaluate using a comprehensive suite of automatic metrics matching the WMT shared task evaluation protocol: BLEU, chrF, ROUGE-L, METEOR, TER, Cosine Similarity, and COMET. All systems use beam search with beam size 4. We evaluate both directions on both test sets... Human evaluation was conducted on 50 en→trp translations... Each translation was rated on two 1–5 scales: Adequacy (does the translation preserve the meaning of the source?) and Fluency (is the translation natural and grammatically correct?)

Citation

@misc{nyalang2026kokborokmt,
  title={Towards High-Quality Machine Translation for Kokborok: A Low-Resource Tibeto-Burman Language of Northeast India},
  author={Nyalang et al. (2026)},
  year={2026},
  note={arXiv:2604.19778}
}

• arXiv: 2604.19778