Codett Eval

codett-eval

CoDeTT: A Context-Aware Decision Benchmark for Turn-Taking Evaluation — Shen et al. (2026) (arXiv:2603.25434, 2026)

What this evaluates

This benchmark evaluates a model's ability to make context-aware turn-taking decisions in multi-turn dialogues. It probes whether models can correctly predict one of four functional actions based on dialogue history and the current system state. The evaluation further diagnoses performance across 14 fine-grained interactional scenarios to reveal semantic misalignments beyond binary end-of-utterance detection.

Datasets

• CoDeTT — total 18000; splits: test (18000); repo https://github.com/YingaoWang-casia/CoDeT.github.io

Metrics

• 4-Action Accuracy (primary) — range: percent
  • Percentage of correctly predicted actions out of the total instances. Predictions are classified into four discrete categories: Maintain, Stop & Listen, Takeover, and Dismiss.
• Semantic Misalignment Rate (SMR) — range: percent
  • Measures the rate of semantic misalignment between predicted and ground-truth decision scenarios, used for diagnostic-level assessment of nuanced interactional intents.

Input / output format

Input: Five complete rounds of multi-turn dialogue history plus the target user query, along with the current system state (SystemSpeaking or SystemIdle).

Output: A single discrete class label from the set: {Maintain, Stop & Listen, Takeover, Dismiss}.

Scoring recipe

def compute_4action_acc(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Models are evaluated separately by input modality (audio vs. text); mixing modalities or ignoring the input type leads to invalid comparisons.
• The 14 fine-grained scenarios must be correctly aggregated to the 4 coarse actions for the headline metric; evaluating only at the scenario level obscures the primary evaluation.
• System state conditioning is mandatory; predictions must be generated separately or conditioned on whether the system is currently Speaking or Idle, as half the dataset (9,000 samples) belongs to each state.

Evidence (verbatim from paper)

CoDeTT is a context-aware decision benchmark for systematic evaluation of turn-taking models. It formalizes turn-taking as a structured decision problem in which models predict one of four functional actions (Maintain, Stop & Listen, Takeover, Dismiss) conditioned on dialogue context and system state. In total, CoDeTT comprises 18,000 annotated decision instances categorized into 14 fine-grained scenarios. Table 2: 4-Action ACC Results: Comparison between Chinese(ZH) and English(EN).

Citation

@misc{shen2026codett,
  title={CoDeTT: A Context-Aware Decision Benchmark for Turn-Taking Evaluation},
  author={Shen et al. (2026)},
  year={2026},
  note={arXiv:2603.25434}
}

• arXiv: 2603.25434