Bhashaverse Translation Eval

bhashaverse-translation-eval

BhashaVerse : Translation Ecosystem for Indian Subcontinent Languages — Mujadia et al. (2024) (arXiv:2412.04351, 2024)

What this evaluates

Evaluates multilingual machine translation and related sequence-to-sequence tasks across 36 Indian subcontinent languages. It probes the model's ability to handle morphological complexity, script diversity, code-mixing, and domain-specific adaptation through reference-based and reference-free metrics.

Datasets

• FLORES + IN22 — total 902198; splits: test (902198)
• Reserved Development Corpora — total 22000; splits: dev (22000)

Metrics

• BLEU (primary) — range: [0, 100]
  • Geometric mean of precisions up to n-grams (typically 4-gram) with a brevity penalty to penalize overly short translations.
• CHRF3 — range: [0, 100]
  • Character n-gram F-score using trigrams, measuring overlap between reference and hypothesis at the character level.
• COMET-22 — range: [0, 1]
  • Reference-based neural metric trained on human judgments, outputting a score between 0 and 1 representing translation quality.
• Spearman Correlation — range: [-1, 1]
  • Rank-based correlation coefficient measuring the monotonic relationship between predicted scores and human judgments.
• F1 Score — range: [0, 1]
  • Harmonic mean of precision and recall for token-level or span-level error identification tasks.

Input / output format

Input: Source sentence (English or Indian subcontinent language) or grammatically incorrect sentence for correction tasks.

Output: Target sentence (translated or corrected output).

Scoring recipe

def compute_metrics(preds, refs):
    bleu = sacrebleu.corpus_bleu(preds, [refs]).score
    chrf = sacrebleu.corpus_chrf(preds, [refs]).score
    f1 = f1_score(refs, preds, average='macro')
    spearman = spearmanr(refs, preds).correlation
    comet = comet_model.predict(preds, refs)
    return {'bleu': bleu, 'chrf3': chrf, 'f1': f1, 'spearman': spearman, 'comet22': comet}

Common pitfalls

• The development corpora for non-MT tasks are explicitly curated from the training data, risking data leakage if treated as independent test sets.
• Average scores are reported across 36 languages with vastly different resource levels, masking performance on low-resource pairs.
• CHRF3 is used instead of the more common CHRF++ or standard CHRF, requiring careful implementation to match the reported scores.

Evidence (verbatim from paper)

The model is evaluated on machine translation using benchmark corpora such as FLORES and IN22, covering available languages, including English and other Indian subcontinent languages. With a combined dataset size of 902,198 sentences, the model achieved an average BLEU score (Papineni et al., [2002]) of 25.45, CHRF3 score (Popović, [2015]) of 53.81, and COMET-22 score (Rei et al., [2022a]) of 0.8138.

Citation

@misc{mujadia2024bhashaverse,
  title={BhashaVerse : Translation Ecosystem for Indian Subcontinent Languages},
  author={Mujadia et al. (2024)},
  year={2024},
  note={arXiv:2412.04351}
}

• arXiv: 2412.04351