Interactive Audio Eval

interactive-audio-eval

Mind the Gap! Static and Interactive Evaluations of Large Audio Models — Li et al. (2025) (arXiv:2502.15919, 2025)

What this evaluates

Evaluates Large Audio Models (LAMs) on real-world, task-oriented voice assistant interactions by capturing user preferences through open-ended pairwise comparisons. It measures how well models align with actual user needs and preferences in an interactive setting, rather than relying on static reference-based benchmarks.

Datasets

• TalkArena Interactive User Preferences — total 7500; splits: full (7500)

Metrics

• Bradley-Terry model score (primary) — range: other
  • A latent preference score derived from pairwise user votes. The probability that model i is preferred over model j is Pr(i>j) = e^{β_i} / (e^{β_i} + e^{β_j}), where β coefficients are optimized by maximizing the log-likelihood of observed preferences. Final scores are computed as p_i = e^{β_i}.

Input / output format

Input: User voice query (streamed character-by-character) + two anonymous model responses presented in random order.

Output: User selects preferred model (i > j, j > i, or tie/no preference) + optional text or speech justification.

Scoring recipe

D = list of (model_i, model_j, y) where y ∈ {1, 0.5, 0}
# Optimize β to maximize log-likelihood
L(β) = Σ [y*log(σ(β_i - β_j)) + (1-y)*log(σ(β_j - β_i))]
β_opt = minimize(-L(β), method='L-BFGS')
scores = {m: exp(β_opt[m]) for m in models}
rank = sorted(models, key=lambda m: scores[m], reverse=True)

Common pitfalls

• Pairwise votes yield relative rankings, not absolute performance scores, making cross-session or cross-pool comparisons invalid without re-estimating the Bradley-Terry coefficients.
• The Bradley-Terry model assumes transitivity of preferences, which may not hold in open-ended, style-driven user interactions.
• Positional and personal biases are mitigated by random ordering and anonymization, but users may still prefer models based on response latency or conversational style rather than factual accuracy.

Evidence (verbatim from paper)

After submitting a query, users receive responses from two anonymous models, which are randomly selected and ordered in order to avoid personal and positional bias in their preferences. For assessment, users provide a simple pairwise preference ranking—choosing the better response or indicating no preference between the two. To convert the collected pairwise preference data to model ranking, we apply the Bradley-Terry model ... The Bradley-Terry coefficients β are computed by maximizing the log-likelihood of the observed pairwise preferences D ... We then compute p_i = e^{β_i}.

Citation

@misc{li2025mindthegap,
  title={Mind the Gap! Static and Interactive Evaluations of Large Audio Models},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2502.15919}
}

• arXiv: 2502.15919