Menaspeechbank Eval

menaspeechbank-eval

MENASpeechBank: A Reference Voice Bank with Persona-Conditioned Multi-Turn Conversations for AudioLLMs — Zien Sheikh Ali et al. (2026) (arXiv:2602.07036, 2026)

What this evaluates

Evaluates AudioLLMs on multi-turn, persona-conditioned spoken dialogue generation. It probes the model's ability to maintain speaker consistency, track conversation context, and generate contextually appropriate text responses to audio inputs in Arabic (MSA) and English.

Datasets

• MENA SpeechBank — total 416297; splits: train (328519), dev (36503), test (51275)

Metrics

• Average Rubric Score (ARS) (primary) — range: [0, 1]
  • Mean pass rate across eight LLM-judged rubrics (relevance, completeness, specificity/actionability, coherence, context tracking, calibration, language/tone match, and safety/appropriateness) applied to the final assistant turn per conversation.
• Average Pass Rate (APR) — range: [0, 1]
  • Fraction of conversation turns where the assistant response satisfies all eight rubric checks jointly, as determined by the LLM-as-a-judge.

Input / output format

Input: Audio user turns and preceding assistant text responses (conversation history), along with persona profile memory and session scenario context.

Output: Text response from the assistant for each user audio turn.

Scoring recipe

rubrics = ['relevance', 'completeness', 'specificity/actionability', 'coherence', 'context_tracking', 'calibration', 'language_tone_match', 'safety_appropriateness']
ars_scores, apr_scores = [], []
for turn in conversation:
    judge_output = llm_judge(turn, history, persona, scenario)
    passes = [judge_output.check(r) for r in rubrics]
    ars_scores.append(sum(passes) / len(rubrics))
    apr_scores.append(1.0 if all(passes) else 0.0)
final_ars = sum(ars_scores) / len(ars_scores)
final_apr = sum(apr_scores) / len(apr_scores)

Common pitfalls

• The LLM-as-a-judge only evaluates the final candidate assistant turn per conversation, not every intermediate turn, which may underrepresent multi-turn consistency.
• Baseline comparisons mix audio-native models with ASR-then-LLM pipelines, making it difficult to isolate transcription errors from generative capability.
• The evaluation subset is small (100 conversations) and recorded by human annotators with ~12% WER, limiting generalizability to fully synthetic or noisy audio.

Evidence (verbatim from paper)

To score generated responses, we use an LLM-as-a-judge that evaluates only the final candidate assistant turn while leveraging the full preceding transcript, the provided profile memory (persona facts available to the assistant), and the session scenario. The judge outputs a structured checklist over eight rubrics: relevance, completeness, specificity/actionability, coherence, context tracking, calibration, language/tone match, and safety/appropriateness. We summarize quality using Average Rubric Score (ARS), computed as the mean pass rate across rubric checks, and Average Pass Rate (APR), computed as the fraction of turns that satisfy the required rubric checks jointly (i.e., an overall pass).

Citation

@misc{ali2026menaspeechbank,
  title={MENASpeechBank: A Reference Voice Bank with Persona-Conditioned Multi-Turn Conversations for AudioLLMs},
  author={Zien Sheikh Ali et al. (2026)},
  year={2026},
  note={arXiv:2602.07036}
}

• arXiv: 2602.07036