Ytseg Segmentation Eval

ytseg-segmentation-eval

From Text Segmentation to Smart Chaptering: A Novel Benchmark for Structuring Video Transcriptions — Retkowski et al. (2024) (arXiv:2402.17633, 2024)

What this evaluates

Evaluates a model's ability to detect topic boundaries in unstructured spoken transcriptions (text segmentation) and generate coherent chapter titles (smart chaptering). It probes hierarchical structuring, real-time/online processing constraints, and cross-domain generalization to meeting transcripts.

Datasets

• WIKI-727K — total ?; splits: test (-1)
• YTSEG — total ?; splits: test (-1)
• QMSUM — total 232; splits: test (-1)
• YTSEG[TITLES] — total ?; splits: test (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall for binary boundary detection: 2 * (P * R) / (P + R).
• Pk — range: [0, 1]
  • Boundary error rate computed over sliding windows; lower values indicate better boundary alignment.
• Boundary Similarity (B1) — range: [0, 1]
  • Measures overlap between predicted and gold boundary sets, penalizing mismatches based on distance.
• ROUGE-1/2/L — range: [0, 1]
  • Lexical overlap metrics measuring unigram, bigram, and longest common subsequence overlap between generated and gold titles.
• BARTScore — range: other
  • Semantic similarity score computed using a fine-tuned BART model; higher values indicate better semantic equivalence.

Input / output format

Input: Sequence of sentences/text segments from video transcriptions or meeting recordings, optionally augmented with previous chapter titles or a fixed future context window (c) for online evaluation.

Output: Binary boundary predictions (segment start/end indices) for segmentation; generated chapter titles for title generation.

Scoring recipe

def score_segmentation(pred_boundaries, gold_boundaries):
    P = len(pred & gold) / len(pred) if pred else 0
    R = len(pred & gold) / len(gold) if gold else 0
    F1 = 2 * P * R / (P + R) if (P + R) else 0
    Pk = compute_pk_error(pred, gold, window=10)
    B1 = compute_boundary_similarity(pred, gold)
    return {'P': P, 'R': R, 'F1': F1, 'Pk': Pk, 'B1': B1}

def score_titles(gen_titles, gold_titles):
    R1 = rouge_score(gen_titles, gold_titles, rouge_types=['rouge1'])
    BS = bartscore_metric(gen_titles, gold_titles)
    return {'R1': R1, 'BS': BS}

Common pitfalls

• QMSUM's small size (232 meetings) makes performance metrics non-robust and cross-dataset transfer results inconclusive.
• Online/real-time evaluation requires careful balancing of future context size (c) and input span (s) to manage latency vs. performance trade-offs.
• Title generation without prior context leads to repetitive functional titles (e.g., 'Intro') and poor stylistic coherence across chapters.

Evidence (verbatim from paper)

We evaluate our segmentation models using a combination of standard binary classification metrics, such as precision, recall, and F1 score, as well as metrics specifically tailored for text segmentation tasks, including Pk as introduced in the work of Beeferman et al. (1999) and Boundary Similarity, as discussed in Fournier (2013). The generation of section titles can be considered an extreme form of summarization. As such, we evaluate our models using established metrics in summarization: ROUGE (Lin, 2004), which measures the lexical overlap, and BARTScore (Yuan et al., 2021), an increasingly used metric for semantic equivalence.

Citation

@misc{retkowski2024smartchaptering,
  title={From Text Segmentation to Smart Chaptering: A Novel Benchmark for Structuring Video Transcriptions},
  author={Retkowski et al. (2024)},
  year={2024},
  note={arXiv:2402.17633}
}

• arXiv: 2402.17633