Biological Visual Eval

biological-visual-eval

BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learning — Jianyang Gu et al. (2025) (arXiv:2505.23883, 2025)

What this evaluates

Evaluates frozen visual embedding extractors on ecological trait alignment, fine-grained intra-species variation preservation, and zero-shot/few-shot transfer learning across diverse biological domains.

Datasets

• FishNet — total 94532; splits: train (75631), test (18901)
• NeWT — total ?; splits: train (-1), test (-1)
• AwA2 — total 37322; splits: train (-1), test (-1)
• Herb — total ?; splits: train (-1), test (-1)
• PlantDoc — total 2598; splits: train (-1), test (-1)
• Life stage-Diff/Align — total ?; splits: train (-1), test (-1)
• Sex-Diff/Align — total 13624; splits: train (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted instances. For FishNet, a sample is counted correct only if all 9 binary labels are predicted correctly.
• average F1 score — range: [0, 1]
  • Macro-averaged F1 score computed across all 85 numeric attributes in AwA2.
• clustering accuracy — range: [0, 1]
  • Accuracy of semi-supervised K-means cluster assignments compared to ground-truth species labels.

Input / output format

Input: Frozen visual embeddings extracted from input images using the evaluated model.

Output: Predicted class labels (binary or multi-class), attribute scores, or cluster assignments.

Scoring recipe

def compute_metric(predictions, gold, metric_name):
    if metric_name == 'accuracy':
        if predictions.ndim > 1:  # FishNet multi-label
            correct = (predictions == gold).all(axis=1)
        else:
            correct = (predictions == gold)
        return correct.mean()
    elif metric_name == 'average F1 score':
        return f1_score(gold, predictions, average='macro')
    elif metric_name == 'clustering accuracy':
        return hungarian_algorithm_accuracy(gold, predictions)
    return 0.0

Common pitfalls

• Fine-tuning the vision model instead of using it as a frozen embedding extractor, which violates the protocol's design to isolate embedding quality.
• Reporting per-label accuracy for FishNet instead of requiring all 9 binary labels to be correct simultaneously for a sample.
• Using the full AwA2 class split instead of the specified 45 train / 5 unseen test zero-shot transfer setting.

Evidence (verbatim from paper)

The average F1 score over all the attributes is reported for this benchmark.

Citation

@misc{gu2025bioclip2,
  title={BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learning},
  author={Jianyang Gu et al. (2025)},
  year={2025},
  note={arXiv:2505.23883}
}

• arXiv: 2505.23883