Transition Based Parsing Eval

transition-based-parsing-eval

Transition-based Parsing with Context Enhancement and Future Reward Reranking — Zhou et al. (2016) (arXiv:1612.05131, 2016)

What this evaluates

This evaluation probes a parser's ability to construct accurate projective dependency trees for sentences. It measures how well the model identifies correct syntactic heads and their grammatical relations (labels) under both English and Chinese linguistic conditions.

Datasets

• Penn Treebank (PTB) v5 — total ?; splits: train (-1), test (-1), dev (-1)
• Chinese Treebank (CTB) v5 — total ?; splits: train (-1), test (-1), dev (-1)

Metrics

• UAS — range: percent
  • The percentage of words with the correct head in the predicted dependency tree.
• LAS (primary) — range: percent
  • The percentage of words with both the correct head and the correct dependency label.

Input / output format

Input: Tokenized sentences with part-of-speech tags (predicted via Stanford POS tagger for English, gold-standard for Chinese) and word segmentation (gold-standard for Chinese).

Output: A projective dependency tree represented as a list of (head_index, dependency_label) pairs for each word in the sentence.

Scoring recipe

def compute_uas_las(pred_deps, gold_deps):
    n = len(gold_deps)
    correct_head = sum(1 for p, g in zip(pred_deps, gold_deps) if p[0] == g[0])
    correct_label = sum(1 for (ph, pl), (gh, gl) in zip(pred_deps, gold_deps) if ph == gh and pl == gl)
    return correct_head / n, correct_label / n

Common pitfalls

• The evaluation strictly enforces projective trees; non-projective structures in the original treebanks must be converted or filtered before scoring.
• English evaluation uses predicted POS tags (~97.2% accuracy) rather than gold tags, which introduces external error sources not present in the Chinese evaluation.
• The Chinese dataset uses gold-standard word segmentation and POS tags, creating an asymmetry in input quality between the two languages.

Evidence (verbatim from paper)

Besides, all experiments are evaluated with unlabeled attachment score (UAS), the percentage of words with the correct head, and labeled attachment score (LAS), the percentage of words with the correct head and label.

Citation

@misc{zhou2016transition,
  title={Transition-based Parsing with Context Enhancement and Future Reward Reranking},
  author={Zhou et al. (2016)},
  year={2016},
  note={arXiv:1612.05131}
}

• arXiv: 1612.05131