Bioscan 5m Eval

bioscan-5m-eval

BIOSCAN-5M: A Multimodal Dataset for Insect Biodiversity — Gharaee et al. (2024) (arXiv:2406.12723, 2024)

What this evaluates

Evaluates models on insect biodiversity monitoring by testing closed-world species identification, open-world genus-level grouping for novel species, and zero-shot clustering of multimodal embeddings against taxonomic ground truth.

Datasets

• BIOSCAN-5M — total 5000000; splits: pretrain (4677756), train (289203), val (14757), test (39373), key_unseen (36465), val_unseen (8819), test_unseen (7887), other_heldout (76590); repo https://github.com/bioscan-m1/BIOSCAN-5M

Metrics

• Fine-tuned accuracy (primary) — range: percent
  • Percentage of correctly predicted species labels on the test split after full model fine-tuning.
• Linear probe accuracy — range: percent
  • Accuracy of a linear classifier trained on frozen pretrained embeddings to predict species labels on the test split.
• 1NN-Probe accuracy — range: percent
  • Genus-level accuracy on unseen species test set using nearest-neighbor classification on averaged token embeddings, fitted on seen species.
• Adjusted Mutual Information (AMI) — range: [0, 1]
  • Clustering agreement score between predicted clusters and ground-truth taxonomic labels, normalized by entropy of true labels.
• Top-1 macro accuracy — range: percent
  • Macro-averaged top-1 accuracy for taxonomic classification across modalities in the multimodal retrieval setting.

Input / output format

Input: DNA barcode sequences (strings), high-resolution insect images, and taxonomic text labels.

Output: Predicted taxonomic labels (species/genus), cluster assignments, or retrieved key indices.

Scoring recipe

def compute_accuracy(preds, gold):
    correct = sum(1 for p, g in zip(preds, gold) if p == g)
    return correct / len(gold) * 100

def compute_1nn_probe(seen_embeds, seen_labels, unseen_seqs, model):
    unseen_preds = []
    for seq in unseen_seqs:
        emb = avg_tokens(model.encode(seq))
        dists = [np.linalg.norm(emb - s) for s in seen_embeds]
        unseen_preds.append(seen_labels[np.argmin(dists)])
    return compute_accuracy(unseen_preds, unseen_labels)

def compute_ami(true_labels, predictions):
    from sklearn.metrics import adjusted_mutual_info_score
    return adjusted_mutual_info_score(true_labels, predictions)

Common pitfalls

• Splits are partitioned by barcode to prevent data leakage across train/val/test sets, meaning all samples sharing a barcode stay together.
• Test set species distribution is flattened to avoid imbalance, unlike the natural dataset distribution.
• Embedding dimensions vary across models (e.g., 128 vs 512 vs 768), which can unfairly impact 1NN probing performance.
• Zero-shot clustering uses UMAP dimensionality reduction to 50D before Agglomerative Clustering, which may obscure fine-grained taxonomic structure.

Evidence (verbatim from paper)

Evaluate against the ground-truth annotations with Adjusted Mutual Information (AMI) score (Vinh et al., 2010), measuring the percentage information explained relative to the entropy of the true labels.

Citation

@misc{gharaee2024bioscan5m,
  title={BIOSCAN-5M: A Multimodal Dataset for Insect Biodiversity},
  author={Gharaee et al. (2024)},
  year={2024},
  note={arXiv:2406.12723}
}

• arXiv: 2406.12723