Hear Eval

hear-eval

HEAR: Holistic Evaluation of Audio Representations — Turian et al. (2022) (arXiv:2203.03022, 2022)

What this evaluates

Evaluates the zero-shot generalization and transferability of pre-trained audio representations across 19 diverse downstream tasks spanning speech, environmental sounds, and music. The benchmark requires models to perform without fine-tuning, emphasizing robustness and cross-domain adaptability.

Datasets

• FSD50K — total ?; splits: test (-1)
• ESC-50 — total ?; splits: test (-1)
• GTZAN — total ?; splits: test (-1)
• Vocal Imitations — total ?; splits: test (-1)
• LibriCount — total ?; splits: test (-1)
• CREMA-D — total ?; splits: test (-1)
• VoxLingua107 — total ?; splits: test (-1)
• Speech Commands — total ?; splits: test (-1)
• DCASE 2016 Task 2 — total ?; splits: test (-1)
• Gunshot Triangulation — total ?; splits: test (-1)
• Beijing Opera — total ?; splits: test (-1)
• Mridingham Stroke and Tonic — total ?; splits: test (-1)
• NSynth — total ?; splits: test (-1)
• Maestro — total ?; splits: test (-1)

Metrics

• normalized score (primary) — range: [-1, 1]
  • Per-task scores are standardized to zero mean and unit variance, then Winsorized (clamped) to [-1, +1] to limit extreme outliers. The final score is the arithmetic mean across all tasks.
• mean top-1 accuracy — range: [0, 1]
  • Standard top-1 classification accuracy averaged across tasks.
• mean average precision (mAP) — range: [0, 1]
  • Mean average precision for multi-label or ranking tasks.

Input / output format

Input: Raw audio clips or segments corresponding to the specific downstream task (e.g., classification, onset detection, pitch estimation, counting).

Output: Task-specific predictions: class labels, counts, pitch values, or temporal onsets, depending on the task.

Scoring recipe

def compute_hear_score(task_scores, task_means, task_stds):
    standardized = {}
    for task, score in task_scores.items():
        standardized[task] = (score - task_means[task]) / task_stds[task]
    winsorized = {t: max(-1.0, min(1.0, s)) for t, s in standardized.items()}
    return sum(winsorized.values()) / len(winsorized)

Common pitfalls

• Evaluating with fine-tuned models instead of the required zero-shot setting.
• Skipping the Winsorization step when aggregating scores, which allows extreme outliers on single tasks to skew inter-task comparisons.
• Relying solely on a single aggregate score, which strips out important task-specific performance nuances highlighted in the full score table.

Evidence (verbatim from paper)

Zhai et al. (2019) compare a variety of aggregation techniques for evaluating cross-task model performance, and find that they are all highly correlated, settling upon simple mean top-1. ... For these summary figures, we normalize each model/task score. Normalized scores allow us to compare models and tasks against each other, under the assumption each task is equally weighted. The normalization procedure is as follows: 1) For each task, we standardize the scores to zero mean and unit variance. Unlike transforming tasks to ranks, we assume that the scale of intra-task scores is important. 2) The standardized scores are Winsorized (clamped) to have variance within $[-1, +1]$ . By limiting the importance of extremely high or low scores on a single task, this approach allows for better inter-task comparison.

Citation

@misc{turian2022hear,
  title={HEAR: Holistic Evaluation of Audio Representations},
  author={Turian et al. (2022)},
  year={2022},
  note={arXiv:2203.03022}
}

• arXiv: 2203.03022