Qualispeech Eval

qualispeech-eval

QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and Descriptions — Siyin Wang et al. (2025) (arXiv:2503.20290, 2025)

What this evaluates

Evaluates auditory large language models' ability to perceive and describe low-level speech quality aspects, including noise, distortion, speed, continuity, listening effort, naturalness, and overall quality. It probes both numerical score prediction and natural language reasoning/description generation.

Datasets

• QualiSpeech — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• PCC (primary) — range: [-1, 1]
  • Pearson Correlation Coefficient measuring the linear correlation between predicted and ground-truth numerical scores across the 7 speech quality aspects.
• Precision — range: [0, 1]
  • Proportion of correctly identified noise, distortion, or unnatural pause instances among all predicted positive instances.
• Recall — range: [0, 1]
  • Proportion of correctly identified noise, distortion, or unnatural pause instances among all actual positive instances.
• IoU — range: [0, 1]
  • Intersection over Union of predicted and ground-truth time intervals for speech impairments, calculated only when the impairment is successfully detected.
• GPT Correlation — range: other
  • Relevance score generated by GPT to gauge the overall alignment and correctness of the model's natural language description against the ground truth.

Input / output format

Input: Audio clip of speech (often containing noise or distortion) plus a multiple-choice scale (1-5) for each of the 7 aspects. For description generation, the audio clip is provided as input.

Output: For the benchmark: a single integer score (1-5) per aspect. For dataset evaluation: a natural language description containing reasoning and specific details about the speech quality.

Scoring recipe

def compute_metrics(predictions, ground_truth):
    # Numerical scores
    pcc = pearsonr(predictions.scores, ground_truth.scores)
    # Detection metrics
    tp = count_correct_detections(predictions, ground_truth)
    fp = count_false_detections(predictions)
    fn = count_missed_detections(ground_truth)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    # Time interval metrics (only if detected)
    iou = iou_score(predictions.intervals, ground_truth.intervals) if recall > 0 else 0
    # Description relevance
    gpt_corr = gpt_evaluate_relevance(predictions.text, ground_truth.text)
    return {'PCC': pcc, 'Precision': precision, 'Recall': recall, 'IoU': iou, 'GPT_Correlation': gpt_corr}

Common pitfalls

• PCC returns NaN when a model predicts the exact same score for all samples in a split.
• IoU and GPT correlation scores are only computed when the model correctly identifies the presence of noise or distortion; undetected samples yield a score of 0.
• The benchmark explicitly excludes open-ended QA due to LLM instruction-following limitations, restricting evaluation to multi-choice score selection.

Evidence (verbatim from paper)

For QualiSpeech benchmark evaluation, PCC (Pearson Correlation Coefficient) is used to evaluate the accuracy of predicted scores. For QualiSpeech dataset evaluation, PCC is also employed to assess aspects represented by numerical values. For aspects in descriptions, we utilize 4 metrics to cover different evaluation dimensions. First, precision and recall are reported to assess the model’s ability to accurately determine the presence of noise (or distortion, unnatural pause). From a complementary point of understanding, correlation scores generated by GPT and intersection over union (IoU) scores are presented.

Citation

@misc{wang2025qualispeech,
  title={QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and Descriptions},
  author={Siyin Wang et al. (2025)},
  year={2025},
  note={arXiv:2503.20290}
}

• arXiv: 2503.20290