Mtr Duplexbench Eval

mtr-duplexbench-eval

MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models — He Zhang et al. (2025) (arXiv:2511.10262, 2025)

What this evaluates

This benchmark evaluates Full-Duplex Speech Language Models (FD-SLMs) on their ability to sustain performance across multi-round conversations. It probes dialogue quality, conversational dynamics (turn-taking, interruptions, pauses, background speech), instruction following, and safety, specifically measuring how these capabilities degrade or hold up as interaction rounds increase.

Datasets

• MTR-DuplexBench — total ?; splits: test (-1)
• Llama Question — total 100; splits: test (100)
• AdvBench — total 100; splits: test (100)

Metrics

• Success Rate (%) (primary) — range: percent
  • Percentage of conversational rounds or feature-handling scenarios where the model successfully produces the expected response behavior without failure.
• Latency (s) — range: other
  • Average time in seconds between user input and model response generation, measured across single or multi-round interactions.
• Backchannel Frequency — range: other
  • Count or frequency of non-intrusive listener responses (e.g., 'uh-huh', nods) produced by the model during conversation rounds.
• GPT-score — range: [0, 5]
  • LLM-as-judge rating of overall dialogue quality on a 5-point scale.
• Refusal Rate (%) — range: percent
  • Percentage of safety-evaluation prompts where the model correctly refuses to generate harmful or unsafe content.

Input / output format

Input: Multi-round audio conversation segments containing specific conversational features (smooth turn-taking, interruptions, pauses, background speech) or instruction/safety prompts, processed turn-by-turn via a turn-segmentation methodology.

Output: Audio responses from the FD-SLM, which are then evaluated for success/failure, latency, backchannel generation, LLM-judge quality scores, or refusal behavior.

Scoring recipe

def compute_metrics(predictions, gold, rounds):
    success_count = 0
    total_latency = 0.0
    backchannel_count = 0
    for i, pred in enumerate(predictions):
        if pred.is_success: success_count += 1
        total_latency += pred.latency_s
        backchannel_count += pred.backchannel_count
    success_rate = (success_count / len(predictions)) * 100
    avg_latency = total_latency / len(predictions)
    # GPT-score and Refusal Rate computed via separate LLM judges on outputs
    return {"success_rate": success_rate, "latency": avg_latency, "backchannel_freq": backchannel_count}

Common pitfalls

• Relying solely on 'first-package latency' instead of reporting multi-round latency, which masks performance degradation over extended conversations.
• Evaluating single conversational features in isolation without testing combined features (e.g., interruptions + background speech), which yields overly optimistic results.
• Ignoring turn segmentation in full-duplex settings, leading to blurred boundaries and inaccurate round-by-round metric aggregation.

Evidence (verbatim from paper)

Across all four conversational features, the success rate of Moshi consistently decreases as the number of interaction rounds increases. This observation is also uniform across the four conversational features and highlights a critical challenge for FD-SLMs, where the communication experience becomes less natural due to higher latency.

Citation

@misc{zhang2025mtrduplexbench,
  title={MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models},
  author={He Zhang et al. (2025)},
  year={2025},
  note={arXiv:2511.10262}
}

• arXiv: 2511.10262