Full Duplex Bench Eval

full-duplex-bench-eval

Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities — Lin et al. (2025) (arXiv:2503.04721, 2025)

What this evaluates

Evaluates real-time interactive behaviors in full-duplex spoken dialogue models. It specifically probes turn-taking, pause handling, backchanneling, and interruption management capabilities without relying on human studies.

Datasets

• Full-Duplex-Bench — total ?; splits: test (-1); repo https://github.com/DanielLin94144/Full-Duplex-Bench

Metrics

• descriptive metrics (primary) — range: other
  • Time-synchronous audio alignment combined with descriptive, reproducible measures to evaluate pause handling, backchanneling, turn-taking, and interruption management.

Input / output format

Input: Audio input processed in 30 ms chunks at 16 kHz PCM-16 format, with server-side VAD for segmentation.

Output: Audio responses aligned with the original input duration, preserving silence in regions where no response is produced.

Scoring recipe

def score(predictions, gold):
    # Align prediction audio with gold input audio time-synchronously
    aligned_pred = align_audio_time_synchronous(predictions, gold)
    # Compute descriptive metrics for turn-taking, pause handling, backchanneling, interruption
    metrics = compute_descriptive_metrics(aligned_pred, gold)
    return metrics

Common pitfalls

• Relying on human studies or dataset-specific training instead of automated, generalizable metrics.
• Ignoring real-time streaming constraints (e.g., 30 ms chunking and server-side VAD) when evaluating model responsiveness.

Evidence (verbatim from paper)

It uses time-synchronous audio alignment and descriptive, reproducible metrics to enable objective, large-scale comparison of models without relying on human studies or dataset-specific training, addressing critical gaps in current evaluation methods that lack generalizability and automation.

Citation

@misc{lin2025fullduplexbench,
  title={Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities},
  author={Lin et al. (2025)},
  year={2025},
  note={arXiv:2503.04721}
}

• arXiv: 2503.04721