Birdset Eval

birdset-eval

BirdSet: A Large-Scale Dataset for Audio Classification in Avian Bioacoustics — Rauch et al. (2024) (arXiv:2403.10380, 2024)

What this evaluates

Evaluates deep learning models on multi-label audio classification for avian bioacoustics, specifically probing robustness to covariate shift, class imbalance, and noisy labels in passive acoustic monitoring scenarios.

Datasets

• BirdSet — total ?; splits: POW (-1), PER (-1), NES (-1), UHH (-1), HSN (-1), NBP (-1), SSW (-1), SNE (-1); repo https://github.com/DBD-research-group/BirdSet

Metrics

• cmAP (primary) — range: [0, 1]
  • Class mean average precision: macro average of Average Precision across all C classes. Formula: (1/C) * sum_{c=1}^C AP(c). Threshold-free metric that ranks positive instances higher than negatives across all thresholds.
• T1-Acc — range: [0, 1]
  • Top-1 Accuracy: fraction of instances where the predicted class with the highest probability is in the set of true labels. Formula: (1/N) * sum_{i=1}^N 1[ŷ_i ∈ Y_i].
• AUROC — range: [0, 1]
  • Area under the receiver operating characteristic curve. Computed as the fraction of positive-negative pairs where the positive score is higher than the negative score. Threshold-independent.

Input / output format

Input: Raw audio waveforms or spectrograms sampled at 16kHz or 32kHz, typically processed into 5-second segments around detected vocalization events.

Output: Multi-label binary classification probabilities or hard labels for a fixed set of bird species classes.

Scoring recipe

def compute_metrics(y_true, y_prob):
    C = y_true.shape[1]
    # cmAP
    aps = []
    for c in range(C):
        precisions, recalls, _ = precision_recall_curve(y_true[:, c], y_prob[:, c])
        aps.append(np.trapz(precisions, recalls))
    cmAP = np.mean(aps)
    # T1-Acc
    preds = np.argmax(y_prob, axis=1)
    true_labels = [np.where(row)[0] for row in y_true]
    t1_acc = np.mean([pred in labels for pred, labels in zip(preds, true_labels)])
    # AUROC
    auroc = roc_auc_score(y_true, y_prob, average='macro')
    return {'cmAP': cmAP, 'T1-Acc': t1_acc, 'AUROC': auroc}

Common pitfalls

• Threshold tuning is explicitly avoided; models are evaluated threshold-free using cmAP and AUROC, which can be noisy for sparse classes.
• The validation split (POW) does not cover all classes in the dedicated training (DT) scenario, requiring a fallback to training splits with augmentations for validation.
• Covariate shift between focal training recordings and soundscape test recordings complicates generalization and requires specific augmentations like background noise mixing and multi-label mixup.

Evidence (verbatim from paper)

We opt for threshold-free metrics to obtain a clear view of overall model performance without the necessity of fine-tuning thresholds for individual classes. This approach enhances comparability and minimizes biases toward particular applications. The metrics implemented are the following: cmAP (class mean average precision) computes the AP (average precision) for each class c as an element independently and then averages these scores across all classes C ... Top-1 Accuracy evaluates whether the class with the highest predicted probability is (one of) the correct class for each instance ... AUROC (area under the receiver operating characteristic curve) computes the area under the receiver operating characteristic curve given a model f

Citation

@misc{rauch2024birdset,
  title={BirdSet: A Large-Scale Dataset for Audio Classification in Avian Bioacoustics},
  author={Rauch et al. (2024)},
  year={2024},
  note={arXiv:2403.10380}
}

• arXiv: 2403.10380