qhjqhj00 speechmentalmanip-eval
Detecting Mental Manipulation in Speech via Synthetic Multi-Speaker Dialogue — Run Chen et al. (arXiv:2601.08342, 2026)
What this evaluates
Binary classification of spoken multi-speaker dialogues to detect the presence of mental manipulation tactics. It probes an audio-language model's ability to identify subtle manipulative cues in synthetic speech without relying on text transcripts.
Datasets
- SpeechMentalManip — total 699; splits: test (699); repo https://github.com/runjchen/speech_mentalmanip
Metrics
accuracy(primary) — range: [0, 1]- Proportion of correctly classified audio clips (manipulative vs. non-manipulative) out of the total evaluation set.
recall— range: [0, 1]- True positive rate for the manipulative class, measuring the fraction of actual manipulative clips correctly identified.
Input / output format
Input: Audio clips of multi-speaker synthetic dialogues generated via a two-phase TTS pipeline.
Output: Binary classification label: 'manipulative' or 'non-manipulative'.
Scoring recipe
def compute_accuracy(preds, gold):
return sum(1 for p, g in zip(preds, gold) if p == g) / len(gold)
def compute_recall(preds, gold, pos='manipulative'):
tp = sum(1 for p, g in zip(preds, gold) if p == pos and g == pos)
actual = sum(1 for g in gold if g == pos)
return tp / actual if actual > 0 else 0.0
Common pitfalls
- Models may exhibit a systematic bias toward predicting 'manipulative' due to safety alignment mechanisms over-flagging agitated prosody, as noted in preliminary trials with other models.
- The dataset uses synthetic TTS audio rather than original movie recordings, which prioritizes experimental control over ecological realism and may affect prosodic cue perception.
Evidence (verbatim from paper)
Prior evaluation on such benchmark indicate that few-shot GPT-4 Turbo reaches 0.724 accuracy and a finetuned LLaMA-2-13B achieves 0.768 accuracy on the core detection task. This process yields 609 manipulative and 90 non-manipulative audio clips used for evaluation.
Citation
@misc{chen2026detecting,
title={Detecting Mental Manipulation in Speech via Synthetic Multi-Speaker Dialogue},
author={Run Chen et al.},
year={2026},
note={arXiv:2601.08342}
}
- arXiv: 2601.08342