Discoeval Eval

discoeval-eval

Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations — Chen et al. (2019) (arXiv:1909.00142, 2019)

What this evaluates

Evaluates whether sentence representations capture discourse-aware semantics by testing performance on tasks involving sentence ordering, discourse relations, and coherence across multiple domains.

Datasets

• DiscoEval — total ?; splits: test (-1); repo https://github.com/ZeweiChu/DiscoEval

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly classified instances. The headline metric is the average accuracy across all DiscoEval tasks.

Input / output format

Input: Sentence embeddings extracted from pretrained encoders (e.g., BERT [CLS] token or ELMo averaged layers), fed into a frozen encoder with a trainable downstream classifier.

Output: Class predictions from the downstream classifier (e.g., sentence order, discourse relation label, coherence score).

Scoring recipe

accuracies = []
for task in discoeval_tasks:
    preds = classifier.encode_and_predict(sentences)
    acc = (preds == gold_labels).mean()
    accuracies.append(acc)
final_score = sum(accuracies) / len(accuracies)

Common pitfalls

• The paper freezes pretrained sentence encoders and only trains a linear classifier, so results reflect representation quality rather than end-to-end fine-tuning.
• DiscoEval aggregates accuracy across heterogeneous tasks (ordering, coherence, discourse relations), which may mask task-specific weaknesses.
• Sentence position and ordering tasks require differentiating neighboring sentences, which can be hurt by losses that encourage similar representations for consecutive sentences.

Evidence (verbatim from paper)

When evaluating on DiscoEval, we encode sentences with pretrained sentence encoders. Following SentEval, we freeze the sentence encoders and only learn the parameters of the downstream classifier. ... 'avg.' is the averaged accuracy for all tasks in DiscoEval.

Citation

@misc{chen2019discoeval,
  title={Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations},
  author={Chen et al. (2019)},
  year={2019},
  note={arXiv:1909.00142}
}

• arXiv: 1909.00142