Rst Discourse Parsing Eval

rst-discourse-parsing-eval

MEGA RST Discourse Treebanks with Structure and Nuclearity from Scalable Distant Sentiment Supervision — Huber et al. (2020) (arXiv:2011.03017, 2020)

What this evaluates

Evaluates a model's ability to predict RST-style discourse tree structure and nuclearity relations between elementary discourse units (EDUs). It probes both intra-domain and inter-domain generalization of discourse parsing across different text genres (news, instructions, reviews).

Datasets

• RST-DT — total ?; splits: train (-1), test (-1)
• Instr-DT — total ?; splits: train (-1), test (-1)
• MEGA-DT — total ?; splits: train (-1)
• Yelp13-DT — total ?; splits: train (-1)

Metrics

• Parseval (Par.) / RST-Parseval (R-Par.) (primary) — range: percent
  • Micro-averaged F1 score for predicted vs. gold discourse relations and nuclearity labels. Original Parseval computes standard precision/recall over relation spans. RST-Parseval (R-Par.) is a constrained variant that only counts predictions valid under RST structural and nuclearity rules.

Input / output format

Input: EDU-segmented text documents (e.g., news articles, instructional guides, or customer reviews).

Output: A hierarchical discourse tree specifying binary relations between EDUs and their nuclearity status (Nucleus-Satellite, Satellite-Nucleus, or Nucleus-Nucleus).

Scoring recipe

def compute_parseval(pred_tree, gold_tree):
    pred_rels = extract_relations_and_nuclearity(pred_tree)
    gold_rels = extract_relations_and_nuclearity(gold_tree)
    tp = len(pred_rels & gold_rels)
    fp = len(pred_rels - gold_rels)
    fn = len(gold_rels - pred_rels)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
# For RST-Parseval, apply RST validity constraints before counting TP/FP/FN

Common pitfalls

• Confusing intra-domain evaluation (train/test on same genre) with inter-domain evaluation (train on one genre, test on another).
• Using Original Parseval vs. RST-Parseval interchangeably; RST-Parseval penalizes structurally invalid RST trees more heavily.
• Failing to average results over multiple stochastic runs; the paper averages over 10 independent runs (or 3 generation processes for stochastic models).

Evidence (verbatim from paper)

The results of the final evaluation are summarized and aggregated in three sets of experiments in Table 3. In the first set (on top of Table 3), we show the micro-averaged original Parseval performance (Par.) (Morey et al., 2017) as well as the RST-Parseval measures (R-Par.) of standard linguistic baselines for the structure- and nuclearity-prediction task.

Citation

@misc{huber2020mega,
  title={MEGA RST Discourse Treebanks with Structure and Nuclearity from Scalable Distant Sentiment Supervision},
  author={Huber et al. (2020)},
  year={2020},
  note={arXiv:2011.03017}
}

• arXiv: 2011.03017