Tree Vs Sequence Eval

tree-vs-sequence-eval

When Are Tree Structures Necessary for Deep Learning of Representations? — Li et al. (2015) (arXiv:1503.00185, 2015)

What this evaluates

This evaluation protocol compares recursive tree-based neural models against recurrent sequence-based models across multiple NLP tasks. It probes whether syntactic tree structures are necessary for learning representations, particularly for tasks requiring long-distance dependency modeling or hierarchical composition.

Datasets

• Stanford Sentiment Treebank — total 215154; splits: test_root (2210), test_phrase (82600)
• Pang Sentiment Dataset — total 10601; splits: train (8101), dev (500), test (2000)
• UMD-QA — total ?; splits: (unstated)
• SemEval-2010 Task 8 — total ?; splits: (unstated)
• Discourse Parsing — total ?; splits: (unstated)

Metrics

• accuracy (primary) — range: [0, 1]
  • Number of correct predictions divided by the total number of evaluation instances. Reported at phrase level, sentence root level, or fine-grained/coarse-grained depending on the task.
• p-value — range: [0, 1]
  • Statistical significance computed via bootstrap resampling. Values below 0.05 indicate a significant difference between model variants.

Input / output format

Input: Sentences with or without parse trees, or clause-segmented sequences. For QA, questions paired with candidate answer phrases. For relation classification, sentences with two marked nominals.

Output: Predicted class labels (e.g., sentiment polarity, semantic relation type, discourse relation) or selected answer phrase from a candidate pool.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

# Statistical significance via bootstrap resampling
# p < 0.05 marked with asterisk (*)

Common pitfalls

• Comparing tree and sequence models without accounting for dataset construction bias (e.g., SST was built on parse trees, potentially favoring tree models).
• Assuming clausal segmentation via punctuation fully replicates the benefits of syntactic parsing.
• Using different training frameworks or hyperparameter tuning protocols across model variants, which can confound performance differences.

Evidence (verbatim from paper)

Models are evaluated at both the phrase level (82,600 instances) and the sentence root level (2,210 instances). ... Table 1: Test set accuracies on the Stanford Sentiment Treebank at root level.

Citation

@misc{li2015tree,
  title={When Are Tree Structures Necessary for Deep Learning of Representations?},
  author={Li et al. (2015)},
  year={2015},
  note={arXiv:1503.00185}
}

• arXiv: 1503.00185