Echochain Eval

echochain-eval

EchoChain: A Full-Duplex Benchmark for State-Update Reasoning Under Interruptions — Modi et al. (2026) (arXiv:2604.16456, 2026)

What this evaluates

Evaluates how voice assistants handle mid-generation interruptions by testing their ability to revise in-progress responses while maintaining context and switching objectives. It probes state-update reasoning, contextual inertia, interruption amnesia, and objective displacement under full-duplex interaction conditions.

Datasets

• EchoChain — total ?; splits: test (-1)

Metrics

• pass_fail (primary) — range: [0, 1]
  • Binary evaluation based on instance-specific rubrics. Calculated as the number of model responses that meet the rubric criteria divided by the total number of evaluated instances.

Input / output format

Input: Full conversation transcripts and aligned synthetic audio, including initial user prompts, mid-generation interruptions, and the model's post-interruption response.

Output: Binary pass/fail label per model response, with failed responses additionally categorized into a predefined failure taxonomy (contextual inertia, interruption amnesia, objective displacement).

Scoring recipe

def compute_pass_fail(predictions, rubrics):
    passes = 0
    for pred, rubric in zip(predictions, rubrics):
        if pred.satisfies(rubric):
            passes += 1
    return passes / len(predictions)

Common pitfalls

• Rubrics are instance-specific and manually authored, introducing potential annotator subjectivity or inconsistency across different conversation contexts.
• Audio quality artifacts from synthetic speech may unfairly penalize models if reviewers conflate generation errors with state-update reasoning failures.
• Blinding is only applied during the rubric application stage; initial screening is done by a single rater per conversation, risking bias before model identity is hidden.

Evidence (verbatim from paper)

Using this rubric, annotators assign a binary pass/fail label for every model response; failed responses that reflect interruption-driven errors are then labeled using the benchmark failure taxonomy.

Citation

@misc{modi2026echochain,
  title={EchoChain: A Full-Duplex Benchmark for State-Update Reasoning Under Interruptions},
  author={Modi et al. (2026)},
  year={2026},
  note={arXiv:2604.16456}
}

• arXiv: 2604.16456