Sagi Eval

sagi-eval

Roadmap towards Superhuman Speech Understanding using Large Language Models — Bu et al. (2024) (arXiv:2410.13268, 2024)

What this evaluates

This benchmark evaluates speech large language models across five hierarchical levels of understanding, ranging from basic automatic speech recognition and language identification to paralinguistic perception (pitch, volume, emotion), abstract acoustic reasoning (medical cough analysis), and creative/agentic tasks (spoken English coaching). It probes the model's ability to process raw audio, follow instructions, and extract both semantic and non-semantic acoustic features.

Datasets

• SAGI Benchmark — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly predicted labels or classifications out of the total evaluation samples. Calculated as (correct predictions / total samples) * 100.
• Word Error Rate (WER) — range: percent
  • Standard ASR metric measuring transcription quality. Calculated as (Substitutions + Deletions + Insertions) / Total Reference Words. Lower values indicate better performance.
• GPT-4 Score — range: other
  • Open-ended tasks (e.g., emotion translation, spoken English coaching) are evaluated by GPT-4 on a discrete 1 to 4 scale based on response quality.

Input / output format

Input: Raw audio input (speech) paired with a text or speech instruction specifying the task (e.g., transcribe, classify emotion, detect pitch).

Output: Text response containing the predicted label, transcription, numerical score, or open-ended answer as specified by the task instruction.

Scoring recipe

def compute_metrics(predictions, gold, task_type):
    if task_type == 'classification':
        return sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold) * 100
    elif task_type == 'asr':
        # Standard WER: (S+D+I)/N * 100
        return calculate_wer(predictions, gold)
    elif task_type == 'gpt4_eval':
        # External LLM judge scores 1-4
        return get_gpt4_score(predictions, gold)
    return None

Common pitfalls

• Models may fail to follow instructions entirely, marked as '×' in results, which should be treated as 0% accuracy or excluded depending on the reporting convention.
• Input instruction modality matters: using text instructions instead of speech instructions significantly boosts performance for some models (e.g., GPT-4o) but degrades it for others (e.g., Qwen2-Audio).
• WER is lower-better, while accuracy and GPT-4 scores are higher-better; mixing these conventions leads to incorrect performance comparisons.

Evidence (verbatim from paper)

“*” denotes that the metric is Word Error Rate (WER) and similar metrics, for which lower values indicate better performance. “†” indicates that the task is evaluated by GPT-4, with a score ranging from 1 to 4.

Citation

@misc{bu2024roadmap,
  title={Roadmap towards Superhuman Speech Understanding using Large Language Models},
  author={Bu et al. (2024)},
  year={2024},
  note={arXiv:2410.13268}
}

• arXiv: 2410.13268