qhjqhj00 comtail-translation-metric
Crosslingual Optimized Metric for Translation Assessment of Indian Languages — Arafat Ahsan et al. (2025) (arXiv:2509.17667, 2025)
What this evaluates
Evaluates the ability of neural models to predict human-assigned translation quality scores for Indian language pairs, measuring alignment with crowd-sourced DA+SQM ratings.
Datasets
- COMTAIL — total 221941; splits: train (199737), dev (9815), test (9887); repo https://github.com/aenaliph/COMTAIL
Metrics
Pearson correlation, Spearman correlation(primary) — range: [-1, 1]- Standard Pearson and Spearman rank correlation coefficients computed between model-predicted quality scores and human-annotated DA+SQM ratings. Ranges from -1 to 1, where 1 indicates perfect agreement.
Input / output format
Input: Source sentence, target/hypothesis sentence, and optionally a reference sentence (for reference-based models).
Output: A single continuous quality score (normalized to [0, 1] or on a 0–100 scale depending on the model variant).
Scoring recipe
def compute_metrics(predictions, human_scores):
pearson = pearsonr(predictions, human_scores)
spearman = spearmanr(predictions, human_scores)
return {'pearson': pearson, 'spearman': spearman}
Common pitfalls
- Source sentences may overlap across train/dev/test splits, potentially causing data leakage.
- Low-resource languages (Kashmiri, Sindhi, Dogri) were excluded from the discernment rater filter due to minimal quality differences in raw data.
- Normalization differs between COMTAIL (custom min-max on z-scores) and WMT datasets (default scikit-learn MinMaxScaler).
Evidence (verbatim from paper)
We create train, dev, test splits on this data in ratio of 0.9, 0.05, 0.05. We group the data by source and target language pairs and randomly sample for each split. Since the data vary greatly across languages pairs this ensures proportional representation in each split. The entire data described in Table [5] are the training, development, and test sets used in building all subsequent models.
Citation
@misc{ahsan2025comtail,
title={Crosslingual Optimized Metric for Translation Assessment of Indian Languages},
author={Arafat Ahsan et al. (2025)},
year={2025},
note={arXiv:2509.17667}
}
- arXiv: 2509.17667