# Kokborok Mt Eval

> Evaluates machine translation quality for Kokborok (a low-resource Tibeto-Burman language) in both English-to-Kokborok and Kokborok-to-English directions. It probes translation adequacy, fluency, and semantic similarity using both automatic metrics and human ratings. Use when the user wants to benchmark on SMOL Test Set, WMT Test Set (Bible domain), or asks about evaluating this task. Reports BLEU.

- Skill: `qhjqhj00/kokborok-mt-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/kokborok-mt-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/kokborok-mt-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/kokborok-mt-eval

---


# kokborok-mt-eval

> Towards High-Quality Machine Translation for Kokborok: A Low-Resource Tibeto-Burman Language of Northeast India — Nyalang et al. (2026) (arXiv:2604.19778, 2026)

## What this evaluates

Evaluates machine translation quality for Kokborok (a low-resource Tibeto-Burman language) in both English-to-Kokborok and Kokborok-to-English directions. It probes translation adequacy, fluency, and semantic similarity using both automatic metrics and human ratings.

## Datasets

- **SMOL Test Set** — total ?; splits: test (-1)
- **WMT Test Set (Bible domain)** — total ?; splits: test (-1)

## Metrics

- `BLEU` **(primary)** — range: percent
  - Corpus-level Bilingual Evaluation Understudy score computed via sacreBLEU with default tokenization. Measures n-gram overlap between system output and reference translations.
- `chrF` — range: percent
  - Character n-gram F-score computed via sacreBLEU. Measures character-level n-gram overlap, robust to morphological variations.
- `ROUGE-L` — range: percent
  - Longest common subsequence F-measure. Measures the longest chain of matching words in order between prediction and reference.
- `METEOR` — range: percent
  - Alignment-based metric using WordNet for synonym matching and stemming. Designed to correlate better with human judgment than BLEU.
- `TER` — range: other
  - Translation Edit Rate. Measures the number of edits (insertions, deletions, substitutions, shifts) needed to change the hypothesis into the reference, divided by reference length.
- `Cosine Similarity` — range: [0, 1]
  - Semantic similarity computed using LaBSE (Language-agnostic BERT Sentence Embeddings) to measure vector space proximity between source and target sentences.
- `COMET` — range: [0, 1]
  - Neural evaluation metric using the Unbabel/wmt22-comet-da model. Predicts translation quality scores based on contextual embeddings of source, hypothesis, and reference.
- `Adequacy` — range: 1-5
  - Human-rated scale (1-5) measuring whether the translation preserves the meaning of the source sentence.
- `Fluency` — range: 1-5
  - Human-rated scale (1-5) measuring whether the translation is natural and grammatically correct in the target language.

## Input / output format

**Input**: Source sentence in English or Kokborok (trp)

**Output**: Translated sentence in Kokborok (trp) or English

## Scoring recipe

```python
def score_mt(predictions, references):
    # Automatic metrics via sacreBLEU
    bleu = sacrebleu.corpus_bleu(predictions, [references]).score
    chrf = sacrebleu.corpus_chrf(predictions, [references]).score
    rouge_l = sacrebleu.corpus_rouge(predictions, [references]).score.fmeasure
    meteor = sacrebleu.corpus_meteor(predictions, [references]).score
    ter = sacrebleu.corpus_ter(predictions, [references]).score  # lower is better
    # Semantic & Neural
    cosine_sim = cosine_similarity(labse_embed(predictions), labse_embed(references))
    comet_score = comet_model.predict(predictions, references)
    return {'BLEU': bleu, 'chrF': chrf, 'ROUGE-L': rouge_l, 'METEOR': meteor, 'TER': ter, 'Cosine': cosine_sim, 'COMET': comet_score}
```

## Common pitfalls

- TER is lower-is-better while all other automatic metrics are higher-is-better, which can cause sign errors in aggregation.
- LaBSE-based Cosine Similarity fails for Kokborok due to the language's absence in LaBSE's training data, making semantic similarity scores unreliable.
- Human evaluation used a small sample (n=50) with mixed expertise, leading to low inter-annotator agreement for the untrained native speaker (κ=0.13).

## Evidence (verbatim from paper)

> We evaluate using a comprehensive suite of automatic metrics matching the WMT shared task evaluation protocol: BLEU, chrF, ROUGE-L, METEOR, TER, Cosine Similarity, and COMET. All systems use beam search with beam size 4. We evaluate both directions on both test sets... Human evaluation was conducted on 50 en→trp translations... Each translation was rated on two 1–5 scales: Adequacy (does the translation preserve the meaning of the source?) and Fluency (is the translation natural and grammatically correct?)

## Citation

```bibtex
@misc{nyalang2026kokborokmt,
  title={Towards High-Quality Machine Translation for Kokborok: A Low-Resource Tibeto-Burman Language of Northeast India},
  author={Nyalang et al. (2026)},
  year={2026},
  note={arXiv:2604.19778}
}
```

- arXiv: 2604.19778