Scielo Parallel Corpus Eval

scielo-parallel-corpus-eval

A Large Parallel Corpus of Full-Text Scientific Articles — Soares et al. (2019) (arXiv:1905.01852, 2019)

What this evaluates

Evaluates the quality of sentence alignment and machine translation performance on a trilingual scientific article corpus. It probes cross-lingual translation accuracy and structural alignment precision in a specialized academic domain.

Datasets

• Scielo Parallel Corpus — total ?; splits: train (-1), tuning (-1), test (-1)

Metrics

• BLEU (primary) — range: percent
  • Standard n-gram precision with brevity penalty, computed using the Moses SMT toolkit on the held-out test set.
• alignment accuracy — range: percent
  • Ratio of manually verified correctly aligned sentence pairs to the total number of sampled sentences, expressed as a percentage.

Input / output format

Input: Source sentence in English, Portuguese, or Spanish.

Output: Target sentence in the corresponding target language (English, Portuguese, or Spanish).

Scoring recipe

# Split corpus per language pair into 85% train, 5% tuning, 10% test
# Train Moses SMT model on train set, tune on tuning set
# Generate translations for test set
bleu_score = moses_bleu(test_source, test_hypotheses)

# Manual validation subset
sampled_sentences = random.sample(corpus_sentences, 400)
correct = count_correct_alignments(sampled_sentences)
alignment_accuracy = (correct / len(sampled_sentences)) * 100

Common pitfalls

• Domain-specific scientific nomenclature and context often cause translation incoherences that BLEU fails to capture accurately.
• Manual alignment validation was only performed on 400 randomly sampled sentences, which may not represent the full corpus distribution.
• The corpus is distributed in TMX format, requiring conversion to standard parallel corpus formats for most modern NLP pipelines.

Evidence (verbatim from paper)

Prior to the SMT experiments, all sentences were randomly split in three disjoint datasets for each language pair: training, tuning and test. Approximately 85% of the aligned sentences were kept for training, 5% for tuning and 10% for test. Table 2 presents the BLEU scores for each language pair for the test set. We manually validated the alignment quality for 400 sentences randomly sampled from the parsed corpus. Figure 2 depicts the rate of correct alignments for each subset of parallel languages.

Citation

@misc{soares2019large,
  title={A Large Parallel Corpus of Full-Text Scientific Articles},
  author={Soares et al. (2019)},
  year={2019},
  note={arXiv:1905.01852}
}

• arXiv: 1905.01852