Isdrama Eval

isdrama-eval

ISDrama: Immersive Spatial Drama Generation through Multimodal Prompting — Yu Zhang et al. (2025) (arXiv:2504.20630, 2025)

What this evaluates

This evaluation protocol assesses the capability of multimodal speech synthesis models to generate high-fidelity, spatially accurate binaural audio from scripts, poses, and prompts. It probes content accuracy, speaker similarity, prosodic expressiveness, and precise spatial localization (interaural phase/level differences and angle/distance consistency).

Datasets

• MRSDrama — total ?; splits: test (-1)

Metrics

• CER — range: percent
  • Character Error Rate measuring the percentage of incorrectly transcribed characters compared to the ground truth script.
• SIM — range: [0, 1]
  • Cosine similarity between speaker embeddings of the generated audio and the reference prompt audio.
• FFE — range: other
  • F0 Frame Error measuring the discrepancy in fundamental frequency frames between generated and ground truth audio.
• IPD MAE (primary) — range: other
  • Mean Absolute Error of Interaural Phase Difference extracted from binaural mel-spectrograms compared to ground truth.
• ILD MAE — range: other
  • Mean Absolute Error of Interaural Level Difference extracted from binaural mel-spectrograms compared to ground truth.
• ANG Cos — range: [0, 1]
  • Cosine similarity of angle embeddings extracted via SPATIAL-AST between generated and ground truth audio.
• DIS Cos — range: [0, 1]
  • Cosine similarity of distance embeddings extracted via SPATIAL-AST between generated and ground truth audio.
• MOS-Q — range: other
  • Mean Opinion Score (1-5) evaluating synthesized quality, naturalness, spatial perception, and coherence.
• MOS-S — range: other
  • Mean Opinion Score (1-5) assessing speaker similarity in timbre and accent.
• MOS-E — range: other
  • Mean Opinion Score (1-5) measuring the expressiveness of semantically aligned prosody.
• MOS-P — range: other
  • Mean Opinion Score (1-5) evaluating pose consistency between multimodal prompts and generated audio.

Input / output format

Input: Scripts, prompt audio, silent video, and geometric poses (3D position, orientation, Doppler-shifted radial velocity). Raw waveforms are processed into 48kHz mel-spectrograms (1024 window, 256 hop, 80 bins).

Output: Generated binaural speech audio samples (single sentences for quality/similarity metrics; concatenated full dramas for spatial coherence metrics).

Scoring recipe

def compute_spatial_metrics(pred_binaural, gt_binaural):
    pred_mel = compute_mel_spectrogram(pred_binaural, sr=48000, n_fft=1024, hop=256, n_mels=80)
    gt_mel = compute_mel_spectrogram(gt_binaural, sr=48000, n_fft=1024, hop=256, n_mels=80)
    ipd_pred, ild_pred = extract_ipd_ild(pred_mel)
    ipd_gt, ild_gt = extract_ipd_ild(gt_mel)
    ipd_mae = np.mean(np.abs(ipd_pred - ipd_gt))
    ild_mae = np.mean(np.abs(ild_pred - ild_gt))
    pred_ang, pred_dis = spatial_ast_embed(pred_mel)
    gt_ang, gt_dis = spatial_ast_embed(gt_mel)
    ang_cos = cosine_similarity(pred_ang, gt_ang)
    dis_cos = cosine_similarity(pred_dis, gt_dis)
    return {'IPD MAE': ipd_mae, 'ILD MAE': ild_mae, 'ANG Cos': ang_cos, 'DIS Cos': dis_cos}

Common pitfalls

• Evaluating single-sentence metrics on concatenated full dramas, or vice versa, as the paper explicitly separates these protocols for fair comparison.
• Applying standard monaural TTS metrics directly to binaural outputs without channel averaging or spatialization, which misrepresents spatial accuracy.
• Confusing MOS-Q (overall quality/spatial perception) with MOS-P (pose consistency) or MOS-E (prosodic expressiveness), which measure distinct aspects.

Evidence (verbatim from paper)

For objective evaluation, we use Character Error Rate (CER) and Cosine Similarity (SIM) to assess the content accuracy and speaker similarity with prompt audio. F0 Frame Error (FFE) is used to evaluate the quality of prosody modeling. Since existing binaural metrics are scarce and not suitable for our one-stage binaural speech generation task, we have designed several new metrics. We extract Interaural Phase Difference (IPD) and Interaural Level Difference (ILD) from the binaural mel-spectrograms and compute MAE with GT. We also compute cosine similarity of angle and distance embedding extracted from SPATIAL-AST [76] with GT for spatial evaluation.

Citation

@misc{zhang2025isdrama,
  title={ISDrama: Immersive Spatial Drama Generation through Multimodal Prompting},
  author={Yu Zhang et al. (2025)},
  year={2025},
  note={arXiv:2504.20630}
}

• arXiv: 2504.20630