Insectset459 Eval

insectset459-eval

InsectSet459: an open dataset of insect sounds for bioacoustic machine learning — Faiß et al. (2025) (arXiv:2503.15074, 2025)

What this evaluates

Evaluates multi-class bioacoustic classification of insect audio recordings into one of 459 species. It probes a model's robustness to severe class imbalance, highly variable sampling rates, and ultrasonic frequency ranges.

Datasets

• InsectSet459 — total 26399; splits: train (15873), validation (5307), test (5219)

Metrics

• F1 score (primary) — range: percent
  • Macro-averaged F1 score computed across all 459 species classes.
• accuracy — range: percent
  • Per-item classification accuracy, calculated as the proportion of correctly predicted audio files.

Input / output format

Input: Raw audio files (WAV or MP3) with highly variable sample rates (8–500 kHz) and durations (typically ~10s, trimmed to 120s max).

Output: Predicted species label from the 459-class taxonomy.

Scoring recipe

def compute_metrics(predictions, gold_labels):
    accuracy = sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
    classes = set(gold_labels)
    f1_scores = []
    for cls in classes:
        tp = sum(p == cls and g == cls for p, g in zip(predictions, gold_labels))
        fp = sum(p == cls and g != cls for p, g in zip(predictions, gold_labels))
        fn = sum(p != cls and g == cls for p, g in zip(predictions, gold_labels))
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    macro_f1 = sum(f1_scores) / len(f1_scores)
    return {'accuracy': accuracy, 'macro_f1': macro_f1}

Common pitfalls

• Severe long-tail class imbalance means accuracy can be misleading; macro-F1 is required to assess rare species performance.
• Standard spectrogram features (e.g., up to 16–22 kHz) discard ultrasonic content, causing systematic underperformance for species calling outside this range.
• Variable sample rates across files require careful resampling or rate-aware processing to avoid spectral distortion or information loss.

Evidence (verbatim from paper)

Table 3. Overall classification performance of classifiers trained on IS459, evaluated on the test set. F1 score is macro-averaged across classes; accuracy is averaged per-item.

Citation

@misc{faiss2025insectset459,
  title={InsectSet459: an open dataset of insect sounds for bioacoustic machine learning},
  author={Faiß et al. (2025)},
  year={2025},
  note={arXiv:2503.15074}
}

• arXiv: 2503.15074