qhjqhj00 pt-en-scientific-abstracts-eval
A Parallel Corpus of Theses and Dissertations Abstracts — Soares et al. (2019) (arXiv:1905.01715, 2019)
What this evaluates
Evaluates the quality of a newly constructed Portuguese-English parallel corpus of scientific abstracts. It probes the effectiveness of automated sentence alignment algorithms and the translation performance of SMT and NMT models on domain-specific academic text.
Datasets
- Theses and Dissertations Abstracts Corpus — total 1289372; splits: train (-1), dev (-1), test (-1)
Metrics
BLEU(primary) — range: percent- Standard n-gram overlap metric between reference and hypothesis translations, reported as a percentage (0-100).
Alignment Accuracy— range: percent- Percentage of manually validated sentence pairs correctly aligned, with categories for correct, partial, and no alignment.
Input / output format
Input: Single sentence in Portuguese or English.
Output: Translated single sentence in the target language.
Scoring recipe
def compute_bleu_score(references, hypothesis):
# Standard BLEU calculation (0-100 scale)
bleu = compute_ngram_overlap(reference=[references], hypothesis=hypothesis) * 100
return bleu
def compute_alignment_accuracy(gold_labels):
total = len(gold_labels)
correct = sum(1 for label in gold_labels if label == 'correct')
return (correct / total) * 100
Common pitfalls
- Domain mismatch: Google Translate scores lower because it is a generic model, not a domain-specific scientific MT system.
- Segmentation errors: Case folding to lowercase before alignment removes capitalization cues, leading to malformed sentences and partial alignment errors.
Evidence (verbatim from paper)
Table 3 presents the BLEU scores for both translation directions with English and Portuguese on the development and test partitions for Moses and OpenNMT models. We also included the scores for Google Translate (GT) as a benchmark of a state-of-the-art system which is widely used. We manually validated the alignment quality for 400 sentences randomly selected from the parsed corpus and assigned quality labels according Section 2.4. From all the evaluated sentences, 82.30% were correctly aligned, while 13.33% were partially aligned, and 4.35% presented no alignment.
Citation
@misc{soares2019parallel,
title={A Parallel Corpus of Theses and Dissertations Abstracts},
author={Soares et al. (2019)},
year={2019},
note={arXiv:1905.01715}
}
- arXiv: 1905.01715