Beans Zero Eval

beans-zero-eval

NatureLM-audio: an Audio-Language Foundation Model for Bioacoustics — David Robinson et al. (arXiv:2411.07186, 2024)

What this evaluates

Evaluates zero-shot generalization of audio-language models on bioacoustic tasks, including species classification, multilabel detection, call-type prediction, lifestage classification, captioning, and individual counting across diverse taxa.

Datasets

• BEANS-Zero — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correct predictions out of total instances. For classification tasks, model outputs are mapped to the closest predefined label using Levenshtein distance before comparison.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall for multilabel detection tasks. Predictions with low character overlap with valid labels are discarded prior to scoring.
• SPIDEr — range: [0, 1]
  • Composite metric combining SPIDER and METEOR scores to evaluate the quality of generated text captions against reference captions.

Input / output format

Input: Audio recording paired with a text prompt. For classification, the prompt asks for the species name. For detection, the prompt either lists up to 10 species options or instructs the model to list all detected species. For captioning and other tasks, it provides task-specific instructions.

Output: Text string containing the predicted species name(s), a list of detected species, or a descriptive caption. Models output 'None' if no species are detected.

Scoring recipe

def score_classification(pred, gold, labels):
    closest = min(labels, key=lambda l: levenshtein(pred, l))
    return int(closest in gold)

def score_detection(pred, gold, valid_labels):
    if char_overlap(pred, valid_labels) < threshold:
        pred = 'None'
    detected = extract_species_list(pred)
    return f1_score(gold, detected)

def score_captioning(pred, refs):
    return spider_score(pred, refs)

Common pitfalls

• Levenshtein distance mapping is optimized for bioacoustic/Latin names and may artificially reduce performance on general audio datasets like ESC50.
• Comparisons with BirdNET and Perch are not fully fair as they were trained on datasets overlapping with the evaluation sets.
• Results across different unseen taxa columns (species, genus, family) should not be directly compared because the number of labels varies.

Evidence (verbatim from paper)

Table[3] shows the main results measured on the BEANS-Zero species classification and detection datasets. We used accuracy for classification, and F1 for detection tasks. The best and the second best metrics are highlighted and underlined per each dataset.

Citation

@misc{robinson2024naturelmaudio,
  title={NatureLM-audio: an Audio-Language Foundation Model for Bioacoustics},
  author={David Robinson et al.},
  year={2024},
  note={arXiv:2411.07186}
}

• arXiv: 2411.07186