Dualturn Turn Taking Eval

dualturn-turn-taking-eval

DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining — Rajaa (2026) (arXiv:2603.08216, 2026)

What this evaluates

This benchmark evaluates a model's ability to predict conversational turn-taking dynamics and agent actions from dual-channel speech audio. It probes the system's capacity to anticipate speech boundaries, detect backchannels, and classify continuous turn-taking states without relying on explicit silence timeouts or external labels.

Datasets

• otoSpeech — total ?; splits: test (113)
• Switchboard — total ?; splits: test (138)

Metrics

• wF1 (primary) — range: [0, 1]
  • Weighted F1 score averaged across five agent action classes (Continue, Backchannel, Turn-Shift, etc.). Computed as the sum of per-class F1 scores multiplied by their respective support (true positive count), divided by the total support.
• AUC avg — range: [0, 1]
  • Average Area Under the Receiver Operating Characteristic Curve for word-level turn prediction across three classes: Continue (C), Backchannel (B), and Turn-Shift (T).
• BC F1 — range: [0, 1]
  • F1 score specifically computed for the Backchannel agent action class, measuring the balance between precision and recall for sparse backchannel events.

Input / output format

Input: Dual-channel raw audio streams (or frozen Mimi codec embeddings) representing two speakers in a conversation.

Output: Per-channel probabilities or discrete predictions for six turn-taking signals (EOT, HOLD, backchannel, speech onset, etc.), which are linearly aggregated into five agent action classes. For word-level evaluation, binary/multi-class predictions for Continue, Backchannel, and Turn-Shift per word.

Scoring recipe

def compute_wF1(gold_actions, pred_actions):
    classes = ['Continue', 'Backchannel', 'Turn-Shift', 'Shift', 'Hold']
    f1s, supports = [], []
    for c in classes:
        tp = sum(1 for g, p in zip(gold_actions, pred_actions) if g == c and p == c)
        fp = sum(1 for g, p in zip(gold_actions, pred_actions) if g != c and p == c)
        fn = sum(1 for g, p in zip(gold_actions, pred_actions) if g == c and p != c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append(f1)
        supports.append(tp + fn)
    return sum(f * s for f, s in zip(f1s, supports)) / sum(supports)

def compute_auc_avg(gold_classes, pred_scores):
    auc_C = roc_auc_score([1 if g == 'C' else 0 for g in gold_classes], pred_scores['C'])
    auc_B = roc_auc_score([1 if g == 'B' else 0 for g in gold_classes], pred_scores['B'])
    auc_T = roc_auc_score([1 if g == 'T' else 0 for g in gold_classes], pred_scores['T'])
    return (auc_C + auc_B + auc_T) / 3

Common pitfalls

• Confusing the 4-second action definition used in the main evaluation with VAP's original 1-second bidirectional event protocol; the paper explicitly tests both to prove gains come from learned representations, not label window adjustments.
• Assuming backchannel (BC) detection is trivial or well-covered by baselines; BC events are highly sparse (<8% of data) and standard models like VAP achieve 0.000 F1 without a dedicated signal.
• Misinterpreting the two-stage training protocol: Stage-1 is purely generative speech pretraining, while Stage-2 drops the generative loss to focus on signal prediction. Adding auxiliary generative loss in Stage-2 actively harms performance by suppressing sparse task learning.

Evidence (verbatim from paper)

DualTurn outperformed VAP on all five agent action classes across both evaluation datasets (wF1 0.633 vs. 0.389 on Switchboard, wF1 0.707 vs. 0.461 on otoSpeech), Table3. Backchannel detection shows the largest gap, where VAP achieves BC F1 = 0.000 because it has no dedicated BC signal*[inoue2025backchannel]*. VAP cannot distinguish BC from CT even with an LR probe on its predicted projection probabilities. Trained entirely on self-supervised labels like VAP, DualTurn achieves BC F1 = 0.349 (chance F1 $\approx$ 0.080) in predicting "the agent should backchannel now".

Citation

@misc{rajaa2026dualturn,
  title={DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining},
  author={Rajaa (2026)},
  year={2026},
  note={arXiv:2603.08216}
}

• arXiv: 2603.08216