Speakersleuth Eval

speakersleuth-eval

SpeakerSleuth: Evaluating Large Audio-Language Models as Judges for Multi-turn Speaker Consistency — Lee et al. (2026) (arXiv:2601.04029, 2026)

What this evaluates

This benchmark evaluates Large Audio-Language Models (LALMs) on their ability to detect, localize, and discriminate speaker inconsistencies in multi-turn dialogues. It specifically probes whether models rely on acoustic cues or are biased toward textual coherence when judging speaker consistency.

Datasets

• SpeakerSleuth — total ?; splits: test (-1)

Metrics

• detection_accuracy (primary) — range: [0, 1]
  • Percentage of dialogues correctly classified as consistent or inconsistent.
• discrimination_accuracy (primary) — range: [0, 1]
  • Percentage of inconsistent dialogues where the correct inconsistent speaker is identified.
• localization_f1 (primary) — range: [0, 1]
  • F1-score for correctly identifying the specific turn(s) containing the speaker inconsistency.

Input / output format

Input: Audio-only input consisting of a multi-turn dialogue, plus a ≥3-second reference audio sample of the target speaker. In ablation settings, textual context of the dialogue may be appended, or the reference audio may be removed.

Output: Model must output a judgment for each dialogue: (1) consistency label (consistent/inconsistent), (2) turn index/indices where inconsistency occurs (for localization), and (3) identity of the inconsistent speaker (for discrimination).

Scoring recipe

def compute_metrics(preds, golds):
    # Detection & Discrimination Accuracy
    det_correct = sum(1 for p, g in zip(preds['det'], golds['det']) if p == g)
    det_acc = det_correct / len(golds['det'])
    
    disc_correct = sum(1 for p, g in zip(preds['disc'], golds['disc']) if p == g)
    disc_acc = disc_correct / len(golds['disc'])
    
    # Localization F1 (turn-level exact match)
    tp = sum(1 for p, g in zip(preds['loc'], golds['loc']) if set(p) == set(g))
    fp = sum(1 for p, g in zip(preds['loc'], golds['loc']) if set(p) != set(g) and len(p) > 0)
    fn = sum(1 for p, g in zip(preds['loc'], golds['loc']) if set(p) != set(g) and len(g) > 0)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    loc_f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return det_acc, disc_acc, loc_f1

Common pitfalls

• Models exhibit strong modality bias, often prioritizing textual coherence over acoustic cues, which artificially inflates accuracy when text context is provided.
• Performance is highly sensitive to the presence of reference audio; removing it causes drastic accuracy drops or unpredictable fluctuations across models.
• Evaluation scenarios (S1, S2, S3) vary significantly in difficulty and structure, leading to non-comparable baseline scores if not reported per scenario.

Evidence (verbatim from paper)

We report accuracy scores for Detection and Discrimination, and F1-scores for Localization as primary metrics. Detailed metric computation methods are provided in Appendix[B.3], along with additional metrics (Precision, Recall, Exact-Match).

Citation

@misc{lee2026speakersleuth,
  title={SpeakerSleuth: Evaluating Large Audio-Language Models as Judges for Multi-turn Speaker Consistency},
  author={Lee et al. (2026)},
  year={2026},
  note={arXiv:2601.04029}
}

• arXiv: 2601.04029