# Filler Word Detection Eval

> Evaluates a model's ability to detect and classify filler words (e.g., 'uh', 'um') in naturalistic speech recordings. It probes temporal localization accuracy and fine-grained acoustic classification under varying evaluation granularities. Use when the user wants to benchmark on PodcastFillers, or asks about evaluating this task. Reports F1.

- Skill: `qhjqhj00/filler-word-detection-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/filler-word-detection-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/filler-word-detection-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/filler-word-detection-eval

---


# filler-word-detection-eval

> Filler Word Detection and Classification: A Dataset and Benchmark — Zhu et al. (2022) (arXiv:2203.15135, 2022)

## What this evaluates

Evaluates a model's ability to detect and classify filler words (e.g., 'uh', 'um') in naturalistic speech recordings. It probes temporal localization accuracy and fine-grained acoustic classification under varying evaluation granularities.

## Datasets

- **PodcastFillers** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `F1` **(primary)** — range: percent
  - Harmonic mean of Precision and Recall: F1 = 2 * (P * R) / (P + R). Reported at both segment-level and event-level granularities.
- `Precision` — range: percent
  - Ratio of correctly detected/classified filler instances to all instances predicted as filler.
- `Recall` — range: percent
  - Ratio of correctly detected/classified filler instances to all ground truth filler instances.

## Input / output format

**Input**: Raw audio recordings processed into acoustic features (wav2vec or log-mel embeddings) and optionally ASR transcriptions.

**Output**: Binary or multi-class classification labels ('uh', 'um', or other) with temporal boundaries at either the segment level or event level.

## Scoring recipe

```python
def compute_f1(preds, gold, granularity='event'):
    tp = sum(1 for p in preds if any(match(p, g, granularity) for g in gold))
    fp = len(preds) - tp
    fn = len(gold) - tp
    p = tp / (tp + fp) if (tp + fp) > 0 else 0
    r = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * p * r / (p + r) if (p + r) > 0 else 0
```

## Common pitfalls

- Evaluation is reported at two different granularities (segment vs. event); results are not directly comparable across granularities.
- AVC-FillerNet relies on ASR output to generate candidates, making its performance dependent on ASR quality, whereas VC-FillerNet is transcription-free.
- VAD threshold significantly impacts recall; a low threshold (0.1) is required to maximize recall without hurting precision.

## Evidence (verbatim from paper)

> Table 2: Segment- and event-based F1 measure (%) results for separately detecting 'uh' and 'um' with our proposed systems.

## Citation

```bibtex
@misc{zhu2022fillerword,
  title={Filler Word Detection and Classification: A Dataset and Benchmark},
  author={Zhu et al. (2022)},
  year={2022},
  note={arXiv:2203.15135}
}
```

- arXiv: 2203.15135