Filler Word Detection Eval

filler-word-detection-eval

Filler Word Detection and Classification: A Dataset and Benchmark — Zhu et al. (2022) (arXiv:2203.15135, 2022)

What this evaluates

Evaluates a model's ability to detect and classify filler words (e.g., 'uh', 'um') in naturalistic speech recordings. It probes temporal localization accuracy and fine-grained acoustic classification under varying evaluation granularities.

Datasets

• PodcastFillers — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• F1 (primary) — range: percent
  • Harmonic mean of Precision and Recall: F1 = 2 * (P * R) / (P + R). Reported at both segment-level and event-level granularities.
• Precision — range: percent
  • Ratio of correctly detected/classified filler instances to all instances predicted as filler.
• Recall — range: percent
  • Ratio of correctly detected/classified filler instances to all ground truth filler instances.

Input / output format

Input: Raw audio recordings processed into acoustic features (wav2vec or log-mel embeddings) and optionally ASR transcriptions.

Output: Binary or multi-class classification labels ('uh', 'um', or other) with temporal boundaries at either the segment level or event level.

Scoring recipe

def compute_f1(preds, gold, granularity='event'):
    tp = sum(1 for p in preds if any(match(p, g, granularity) for g in gold))
    fp = len(preds) - tp
    fn = len(gold) - tp
    p = tp / (tp + fp) if (tp + fp) > 0 else 0
    r = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * p * r / (p + r) if (p + r) > 0 else 0

Common pitfalls

• Evaluation is reported at two different granularities (segment vs. event); results are not directly comparable across granularities.
• AVC-FillerNet relies on ASR output to generate candidates, making its performance dependent on ASR quality, whereas VC-FillerNet is transcription-free.
• VAD threshold significantly impacts recall; a low threshold (0.1) is required to maximize recall without hurting precision.

Evidence (verbatim from paper)

Table 2: Segment- and event-based F1 measure (%) results for separately detecting 'uh' and 'um' with our proposed systems.

Citation

@misc{zhu2022fillerword,
  title={Filler Word Detection and Classification: A Dataset and Benchmark},
  author={Zhu et al. (2022)},
  year={2022},
  note={arXiv:2203.15135}
}

• arXiv: 2203.15135