Dialseg711 Seg Eval

dialseg711-seg-eval

S3-DST: Structured Open-Domain Dialogue Segmentation and State Tracking in the Era of LLMs — Sarkar Snigdha Sarathi Das et al. (2023) (arXiv:2309.08827, 2023)

What this evaluates

Evaluates dialogue segmentation on a benchmark constructed by joining disparate task-oriented dialogues. It probes the model's ability to detect abrupt, artificial context shifts and identify segment boundaries in synthetic multi-intent conversations.

Datasets

• DialSeg711 — total 711; splits: test (711)

Metrics

• Pk (primary) — range: [0, 1]
  • An error metric quantifying the difference between predicted and ground-truth segment boundaries using an adjustable sliding window. Lower is better.
• WindowDiff — range: [0, 1]
  • An error metric similar to Pk that measures boundary prediction error over a sliding window. Lower is better.

Input / output format

Input: Dialogue turns provided as text, prompted for segmentation labels in a structured format.

Output: Structured predictions of segment boundaries for each turn.

Scoring recipe

def compute_pk_error(pred_boundaries, gold_boundaries, window_size=10):
    errors = 0
    total_windows = 0
    for i in range(len(pred_boundaries) - window_size + 1):
        pred_slice = pred_boundaries[i:i+window_size]
        gold_slice = gold_boundaries[i:i+window_size]
        if sum(pred_slice) != sum(gold_slice):
            errors += 1
        total_windows += 1
    return errors / total_windows

Common pitfalls

• The dataset's construction (joining unrelated dialogues) creates artificial and abrupt context shifts, making it potentially too easy for modern LLMs and not reflective of natural conversation flow.
• LLMs may fail to track original context when segments are joined, leading to high error if grounding mechanisms like PAR are omitted.

Evidence (verbatim from paper)

For segmentation, we consider Pk and WindowDiff (Pevzner and Hearst, 2002), which are both error metrics (i.e., lower is better) that quantify the difference between predicted and ground-truth segment boundaries using an adjustable sliding window. ... S3-DST achieves nearly zero error on this dataset...

Citation

@misc{das2023s3dst,
  title={S3-DST: Structured Open-Domain Dialogue Segmentation and State Tracking in the Era of LLMs},
  author={Sarkar Snigdha Sarathi Das et al. (2023)},
  year={2023},
  note={arXiv:2309.08827}
}

• arXiv: 2309.08827