# Animalclap Taxonomy Aware Pretraining

> Build taxonomy-aware audio-text pretraining systems for species recognition from animal vocalizations. Train contrastive models that augment text prompts with hierarchical taxonomic structure (scientific/common names, phylogenetic sequences), evaluate on unseen species via rare-species test sets, and predict ecological traits directly from audio.

- Skill: `adu2021/animalclap-taxonomy-aware-pretraining` (Agent Skill)
- Install (CLI): `npx skillmds@latest add adu2021/animalclap-taxonomy-aware-pretraining`
- Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/animalclap-taxonomy-aware-pretraining/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: MIT
- Author: adu2021 (https://skillmd.com/u/adu2021)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/adu2021/animalclap-taxonomy-aware-pretraining

---


# AnimalCLAP: Taxonomy-Aware Language-Audio Pretraining

## Problem Statement

Traditional audio classification models fail to recognize unseen animal species because they lack awareness of biological relationships. Audio-only systems (e.g., baseline CLAP) achieve just 1.61% top-1 accuracy on unseen species, while species form natural hierarchies (6 classes → 66 orders → 341 families → 2,152 genera) that could guide learning.

## Core Innovation: Taxonomy-Aware Pretraining

Integrate hierarchical biological structure into contrastive language-audio pretraining by augmenting text representations with taxonomic metadata.

**Data Structure:** 4,225 hours of recordings covering 6,823 species with 22 ecological trait annotations (diet, activity pattern, habitat, climate distribution, social behavior).

**Text Augmentation Strategy:** For each species, generate prompts combining:
- Common name (e.g., "American Robin")
- Scientific name (e.g., *Turdus migratorius*)
- Taxonomic sequence (e.g., "Aves → Passeriformes → Turdidae → Turdus → migratorius")

**Architecture:** HTS-AT audio encoder + RoBERTa text encoder with contrastive loss.

## Key Results

**Unseen Species Classification:** AnimalCLAP achieves 27.6% top-1 accuracy vs. baseline CLAP's 1.61%—a 17× improvement. This validates that hierarchical structure generalizes to novel species.

**Taxonomy Ablation:** Randomizing taxonomic sequence ordering reduces accuracy substantially, confirming that hierarchy ordering (not just presence of names) drives generalization.

**Trait Prediction:** Model successfully predicts ecological traits directly from audio:
- Activity patterns: 83.7% accuracy
- Predator classification: 92.6% accuracy
- Broader environmental traits: lower but significant

**Test Set Design:** 300 rare species with <15 recordings each prevents training data leakage and validates true out-of-distribution generalization.

## Deployment Recipe

1. **Data Collection Criteria:** Use only Creative Commons-licensed recordings from iNaturalist and Xeno-canto; verify habitat/temporal diversity.

2. **Annotation Workflow:** Obtain species labels, aggregate taxonomic names via open databases (NCBI Taxonomy), compute taxonomic sequences programmatically.

3. **Text Prompt Construction:** For species S, generate: "A recording of [common_name], scientifically known as [scientific_name], belonging to the sequence [path_from_class_to_species]."

4. **Training:** Contrastive learning with standard CLIP objective; ensure balanced sampling across taxonomic levels to prevent genus/family overfitting.

5. **Evaluation:** Always construct test sets using rare species (<15 recordings) to measure generalization to unseen taxa. Include both accuracy and trait prediction metrics.

## Practical Implications

- **Scaling:** Hierarchy enables few-shot learning for data-scarce species; economic impact for conservation monitoring.
- **Trait Transfer:** Learned audio representations capture ecological properties, enabling downstream tasks (habitat prediction, behavior classification) without additional annotation.
- **Generalization Principle:** Metadata-informed contrastive learning outperforms brute-force scaling in domains with natural hierarchies.