Cognition Eval

cognition-eval

On Compositional Generalization of Neural Machine Translation — Li et al. (2021) (arXiv:2105.14802, 2021)

What this evaluates

This benchmark evaluates a neural machine translation model's ability to generalize compositionally by translating novel compound phrases that were not seen during training. It probes whether the model can correctly assemble semantic components in new syntactic contexts, revealing gaps between standard sentence-level fluency metrics and actual compositional robustness.

Datasets

• CoGnition — total 227046; splits: train (196246), val (10000), random-test (10000), CG-test (10800); repo https://github.com/yafuly/CoGnition

Metrics

• character-level BLEU — range: percent
  • Computed using SacreBLEU at the character level to measure overall translation performance and fluency across the test set.
• compound translation error rate (primary) — range: percent
  • Human-annotated percentage indicating whether a compound translation contains the semantic meaning of all its constituent atoms and is fluent. Instance rate measures the percentage of compounds translated incorrectly; aggregate rate measures the percentage of compounds with at least one incorrect translation across 5 contexts.

Input / output format

Input: English source sentence (tokenized with Moses tokenizer, vocabulary size 2000)

Output: Chinese target sentence (segmented with jieba, BPE vocabulary size 5500)

Scoring recipe

bleu = sacrebleu.corpus_bleu(predictions, references, tokenize='char')

compound_errors = []
for compound in compounds:
    contexts = get_contexts(compound) # 5 contexts per compound
    correct_in_any = False
    for ctx in contexts:
        if is_compound_correct(ctx.prediction, ctx.ground_truth):
            correct_in_any = True
    compound_errors.append(not correct_in_any)

aggregate_error_rate = sum(compound_errors) / len(compound_errors) * 100
instance_error_rate = sum(not correct_in_any for each instance) / total_instances * 100

Common pitfalls

• Relying solely on sentence-level BLEU scores, which can be misleadingly high even when novel compounds are mistranslated.
• Assuming that high frequency of individual atoms in training guarantees correct translation of unseen compound combinations.
• Using automatic metrics for compound correctness instead of the required human expert annotation focusing strictly on the compound's semantic completeness and fluency.

Evidence (verbatim from paper)

We report character-level BLEU scores using SacreBLEU (Post, 2018) to measure the overall translation performance. In addition, we request expert translators to annotate the correctness of compound translation. Translators are asked to only focus on examining whether the compound itself is translated correctly or not, disregarding errors in context. Specifically, a compound is correct only if its translation contains semantic meaning of all atoms and is fluent in human language. Since each of the 2,160 compounds is provided with 5 contexts, we can compute the translation error-rate for each compound.

Citation

@misc{li2021compositional,
  title={On Compositional Generalization of Neural Machine Translation},
  author={Li et al. (2021)},
  year={2021},
  note={arXiv:2105.14802}
}

• arXiv: 2105.14802