# Discoeval Eval

> Evaluates whether sentence representations capture discourse-aware semantics by testing performance on tasks involving sentence ordering, discourse relations, and coherence across multiple domains. Use when the user wants to benchmark on DiscoEval, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/discoeval-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/discoeval-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/discoeval-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/discoeval-eval

---


# discoeval-eval

> Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations — Chen et al. (2019) (arXiv:1909.00142, 2019)

## What this evaluates

Evaluates whether sentence representations capture discourse-aware semantics by testing performance on tasks involving sentence ordering, discourse relations, and coherence across multiple domains.

## Datasets

- **DiscoEval** — total ?; splits: test (-1); repo https://github.com/ZeweiChu/DiscoEval

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly classified instances. The headline metric is the average accuracy across all DiscoEval tasks.

## Input / output format

**Input**: Sentence embeddings extracted from pretrained encoders (e.g., BERT [CLS] token or ELMo averaged layers), fed into a frozen encoder with a trainable downstream classifier.

**Output**: Class predictions from the downstream classifier (e.g., sentence order, discourse relation label, coherence score).

## Scoring recipe

```python
accuracies = []
for task in discoeval_tasks:
    preds = classifier.encode_and_predict(sentences)
    acc = (preds == gold_labels).mean()
    accuracies.append(acc)
final_score = sum(accuracies) / len(accuracies)
```

## Common pitfalls

- The paper freezes pretrained sentence encoders and only trains a linear classifier, so results reflect representation quality rather than end-to-end fine-tuning.
- DiscoEval aggregates accuracy across heterogeneous tasks (ordering, coherence, discourse relations), which may mask task-specific weaknesses.
- Sentence position and ordering tasks require differentiating neighboring sentences, which can be hurt by losses that encourage similar representations for consecutive sentences.

## Evidence (verbatim from paper)

> When evaluating on DiscoEval, we encode sentences with pretrained sentence encoders. Following SentEval, we freeze the sentence encoders and only learn the parameters of the downstream classifier. ... 'avg.' is the averaged accuracy for all tasks in DiscoEval.

## Citation

```bibtex
@misc{chen2019discoeval,
  title={Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations},
  author={Chen et al. (2019)},
  year={2019},
  note={arXiv:1909.00142}
}
```

- arXiv: 1909.00142