Disordered Dabs Eval

disordered-dabs-eval

Disordered-DABS: A Benchmark for Dynamic Aspect-Based Summarization in Disordered Texts — Guo et al. (2024) (arXiv:2402.10554, 2024)

What this evaluates

Probes a model's ability to perform dynamic aspect-based summarization on disordered, non-sequential texts where sentences from multiple sources are shuffled. It tests whether the model can cluster fragmented content by underlying topics or aspects and generate precise, coherent summaries without relying on original sentence order.

Datasets

• D-CnnDM — total 52022; splits: train (47920), valid (2185), test (1917)
• D-WikiHow — total 203264; splits: train (142284), valid (20327), test (40653)

Metrics

• Human Evaluation (Coherence, Consistency, Fluency, Relevance, Aspect Quality) (primary) — range: 1-5 scale
  • Average rating on a 1–5 scale across five criteria: Coherence, Consistency, Fluency, Relevance, and Aspect Quality. Ratings are provided by three annotators per sample and averaged.

Input / output format

Input: Disordered text composed of shuffled sentences aggregated from multiple source articles or paragraphs, where each source represents a distinct aspect.

Output: Aspect-based summaries corresponding to the identified aspects in the disordered input.

Scoring recipe

def compute_human_eval(predictions, gold, annotators=3):
    criteria = ['Coherence', 'Consistency', 'Fluency', 'Relevance', 'Aspect Quality']
    scores = []
    for sample in predictions:
        annotator_ratings = []
        for _ in range(annotators):
            for criterion in criteria:
                annotator_ratings.append(annotate(sample, criterion, scale=5))
        scores.append(mean(annotator_ratings))
    return mean(scores)

Common pitfalls

• Assuming standard automatic metrics (e.g., ROUGE) are the primary evaluation; the benchmark explicitly relies on human evaluation for quality assessment.
• Overlooking that aspect boundaries can be vague or overlapping, particularly in D-WikiHow, which leads to lower Aspect Quality scores and requires careful annotator calibration.
• Expecting sequential input cues; the intentional sentence shuffling removes positional coherence, significantly impacting both model performance and human comprehension.

Evidence (verbatim from paper)

We conducted a human evaluation of the datasets, adhering to the methodology described in Section 3.2. For this assessment, we randomly selected thirty samples from each of D-CnnDM and D-WikiHow. Each sample was rated by three annotators on a scale from 1 to 5, across five distinct metrics. The results, presented in Table 3, confirm the high quality of both datasets. Notably, the scores for “Coherence” and “Fluency” are impressive... Regarding “Aspect Quality”, D-CnnDM showcases more clearly defined and distinct aspects compared to D-WikiHow.

Citation

@misc{guo2024disordereddabs,
  title={Disordered-DABS: A Benchmark for Dynamic Aspect-Based Summarization in Disordered Texts},
  author={Guo et al. (2024)},
  year={2024},
  note={arXiv:2402.10554}
}

• arXiv: 2402.10554