Audiorole Eval

audiorole-eval

AudioRole: An Audio Dataset for Character Role-Playing in Large Language Models — Li et al. (2025) (arXiv:2509.23435, 2025)

What this evaluates

Evaluates a model's ability to generate audio responses that simultaneously maintain contextual semantic appropriateness and preserve the specific acoustic identity (timbre, prosody) of a target character given reference audio.

Datasets

• AudioRole-Demo — total 11328; splits: train (10704), test (624)

Metrics

• Acoustic Quality (AQ) — range: [0, 10]
  • Score from 0 to 10 computed by Audiobox’s pre-trained aesthetic scoring model, quantifying signal-to-noise ratio, harmonic-to-noise ratio, and spectral flatness.
• Content Quality (CQ) — range: [0, 2]
  • Score from 0 to 2 obtained by transcribing the output audio with whisper-turbo and evaluating semantic alignment/contextual appropriateness using GPT-4o.
• Acoustic Personalization (AP) (primary) — range: other
  • Cosine distance $D_{\text{cos}}$ between speaker embeddings of the reference audio and synthesized output, extracted via PyAnnotate’s wespeaker-voxceleb-resnet34-LM model. Lower values indicate better preservation.
• Content Personalization (CP) — range: [0, 2]
  • Score from 0 to 2 generated by GPT-4o-audio, which analyzes whether the synthesized audio and reference audio exhibit the same character’s stylistic consistency.

Input / output format

Input: Reference audio of the target character ($X_c$) and an input audio prompt ($X_a$).

Output: Synthesized audio response ($X_b$) matching the prompt and character identity.

Scoring recipe

def evaluate(X_c, X_a, X_b):
    aq = audiobox_aesthetic_score(X_b)  # [0, 10]
    text = whisper_turbo.transcribe(X_b)
    cq = gpt4o_semantic_alignment_score(text, context)  # [0, 2]
    ref_emb = pyannote_wespeaker_embedding(X_c)
    gen_emb = pyannote_wespeaker_embedding(X_b)
    ap = cosine_distance(ref_emb, gen_emb)  # lower is better
    cp = gpt4o_audio_style_score(X_b, X_c)  # [0, 2]
    return {'AQ': aq, 'CQ': cq, 'AP': ap, 'CP': cp}

Common pitfalls

• Acoustic Personalization (AP) reports cosine distance where lower values indicate better performance, which is the inverse of typical similarity scores.
• Content Quality (CQ) and Content Personalization (CP) rely on LLM judges (GPT-4o), making results sensitive to prompt phrasing and non-deterministic sampling.
• The test split is highly imbalanced across characters (e.g., 22 turns for Tyrion vs. 250 for Sheldon), so macro-averaging may skew overall results.

Evidence (verbatim from paper)

Acoustic Personalization (AP) quantifies voice characteristic preservation through PyAnnotate’s speaker embedding model111111[https://huggingface.co/pyannote/wespeaker-voxceleb-resnet34-LM ""]. Given reference audio samples $X_{c}$ of $C$ and synthesized $X_{b}$, we calculate voice similarity as $D_{\text{cos}}$. Lower $D_{\text{cos}}$ values indicate better preservation of $C$’s vocal timbre and prosodic patterns. Content Personalization (CP) evaluates stylistic consistency using GPT-4o-audio multi-modal reasoning ability. The model takes in $X_{b}$ and a reference audio sample $X_{c}$, and a prompt to analyze whether they show the same character’s style in a range of 0 to 2.

Citation

@misc{li2025audiorole,
  title={AudioRole: An Audio Dataset for Character Role-Playing in Large Language Models},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2509.23435}
}

• arXiv: 2509.23435