Beans Eval

beans-eval

BEANS: The Benchmark of Animal Sounds — Hagiwara et al. (2022) (arXiv:2210.12300, 2022)

What this evaluates

This benchmark evaluates machine learning models on bioacoustic animal sound recognition across 12 diverse datasets spanning birds, mammals, anurans, and insects. It probes two core capabilities: multi-label species classification and temporal sound detection, testing models' ability to generalize across species and handle varying recording conditions and class imbalances.

Datasets

• wtkn — total ?; splits: train (-1), val (-1), test (-1)
• bat — total ?; splits: train (-1), val (-1), test (-1)
• cbi — total ?; splits: train (-1), val (-1), test (-1)
• hbdb — total ?; splits: train (-1), val (-1), test (-1)
• dogs — total ?; splits: train (-1), val (-1), test (-1)
• dcase — total ?; splits: train (-1), val (-1), test (-1)
• enabirds — total ?; splits: train (-1), val (-1), test (-1)
• hiceas — total ?; splits: train (-1), val (-1), test (-1)
• rfcx — total ?; splits: train (-1), val (-1), test (-1)
• hainan-gibbons — total ?; splits: train (-1), val (-1), test (-1)
• esc — total ?; splits: train (-1), val (-1), test (-1)
• sc — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly classified instances out of the total number of instances. Computed per dataset for classification and auxiliary tasks.
• mean average precision (mAP) — range: [0, 1]
  • Mean of the average precision scores across all classes for detection tasks. An instance is marked positive if the overlap with any ground truth annotation exceeds 20%.

Input / output format

Input: Audio waveforms converted to mono, 16-bit depth. For classification, padded or truncated to dataset-specific duration thresholds. For detection, partitioned into sliding windows (2s for dcase/enabirds, 10s for hiceas/rfcx, 4s for hainan-gibbons). Input features are power mel-spectrograms (16kHz, 50ms FFT, 10ms hop) or 80-dimensional MFCC summary statistics (mean, std, min, max per MFCC dimension).

Output: Classification: discrete species label via softmax layer. Detection: binary presence/absence score per sliding window via sigmoid layer.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

def compute_mAP(predictions, gold_labels):
    # predictions: list of confidence scores, gold_labels: list of 0/1
    # Sort predictions descending by score
    sorted_pairs = sorted(zip(predictions, gold_labels), key=lambda x: x[0], reverse=True)
    tp, fp = 0, 0
    precisions = []
    for score, label in sorted_pairs:
        if label == 1: tp += 1
        else: fp += 1
        precisions.append(tp / (tp + fp))
    # Compute AP per class using trapezoidal rule, then average across classes
    return sum(precisions) / len(precisions)

Common pitfalls

• Padding short waveforms with silence can artificially inflate classification performance if instance lengths differ systematically between classes.
• Detection tasks suffer from extreme sparsity, meaning only a minor portion of recordings contain vocalizations and training annotations per class are very few.
• Positive labeling for detection strictly depends on a 20% overlap threshold between the sliding window and ground truth annotations.

Evidence (verbatim from paper)

As measure for task performance, we used accuracy for classification and auxiliary tasks, and mean average precision (mAP) for detection tasks. An instance is marked positive if the amount of overlap with any annotation is more than 20%.

Citation

@misc{hagiwara2022beans,
  title={BEANS: The Benchmark of Animal Sounds},
  author={Hagiwara et al. (2022)},
  year={2022},
  note={arXiv:2210.12300}
}

• arXiv: 2210.12300