Gwlans Eval

gwlans-eval

GWLAN: General Word-Level AutocompletioN for Computer-Aided Translation — Huayang Li et al. (2021) (arXiv:2105.14913, 2021)

What this evaluates

Predicts target words in computer-aided translation based on source sentences, translation context (prefix, suffix, zero, bidirectional), and human-typed characters. It probes the model's ability to handle discontinuous context and weak positional information in real-world CAT scenarios.

Datasets

• GWLAN Benchmark — total ?; splits: train (1250000), val (-1), test (-1)

Metrics

• accuracy (primary) — range: percent
  • Exact match accuracy: the percentage of instances where the predicted target word exactly matches the gold target word. Scores are averaged across four translation context types (prefix, suffix, zero-context, and bidirectional context).

Input / output format

Input: Source sentence, translation context type (prefix, suffix, zero, or bidirectional), and human-typed character sequence.

Output: A single predicted target word.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return (correct / len(golds)) * 100
# Final reported score is the average of accuracy computed separately for each of the four context types.

Common pitfalls

• The evaluation averages accuracy across four distinct context types (prefix, suffix, zero, bidirectional); reporting only one context's score misrepresents overall performance.
• Trivial instances (e.g., predicting 'the' given 'th') are filtered out using length rules (>4 chars English, >2 chars Chinese) and context length constraints; ignoring these rules inflates scores on easy cases.

Evidence (verbatim from paper)

Table 1: The main results of different systems on Chinese-English and German-English datasets. The results in this table are the averaged accuracy on four translation contexts (i.e., prefix, suffix, zero-context, and bi-context).

Citation

@misc{li2021gwlans,
  title={GWLAN: General Word-Level AutocompletioN for Computer-Aided Translation},
  author={Huayang Li et al. (2021)},
  year={2021},
  note={arXiv:2105.14913}
}

• arXiv: 2105.14913