Biocap Eval

biocap-eval

BioCAP: Exploiting Synthetic Captions Beyond Labels in Biological Foundation Models — Ziheng Zhang et al. (arXiv:2510.20095, 2025)

What this evaluates

Evaluates zero-shot species classification and fine-grained text-image retrieval capabilities in biological domains. Probes the model's ability to align visual features with taxonomic labels and descriptive natural language without task-specific fine-tuning.

Datasets

• NABirds — total ?; splits: test (-1)
• Meta-Album (Plankton, Insects, Insects 2) — total ?; splits: test (-1)
• IDLE-OO Camera Traps — total ?; splits: test (-1)
• Rare Species — total ?; splits: test (-1)
• PlantNet — total ?; splits: test (-1)
• Fungi — total ?; splits: test (-1)
• PlantVillage — total ?; splits: test (-1)
• Med. Leaf — total ?; splits: test (-1)
• INQUIRE-Rerank — total ?; splits: test (-1)
• Cornell Bird — total ?; splits: test (-1)
• PlantID — total ?; splits: test (-1)

Metrics

• top-1 accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted species labels out of the total number of test instances.
• AP@50 — range: [0, 1]
  • Average Precision computed over the top 50 ranked documents for each query.
• Recall@10 — range: [0, 1]
  • Fraction of queries where the ground-truth matching image or text appears in the top 10 retrieved results.

Input / output format

Input: RGB image paired with a species name (for classification) or a natural language query (for retrieval).

Output: Predicted class label (classification) or a ranked list of images/texts (retrieval).

Scoring recipe

def compute_metrics(predictions, golds, k=10, max_docs=50):
    # Classification
    acc = sum(p == g for p, g in zip(predictions['cls'], golds['cls'])) / len(golds['cls'])
    
    # Retrieval Recall@K
    recalls = []
    for img, texts, gold_text in zip(predictions['img'], golds['texts'], golds['gold_text']):
        scores = model.compute_similarity(img, texts)
        ranked = argsort(scores, descending=True)[:k]
        recalls.append(any(gold_text == texts[i] for i in ranked))
    recall_at_k = sum(recalls) / len(recalls)
    
    # AP@50
    aps = []
    for query, docs, gold_doc in zip(predictions['query'], golds['docs'], golds['gold_doc']):
        scores = model.compute_similarity(query, docs)
        ranked = argsort(scores, descending=True)[:max_docs]
        relevant = [1 if docs[i] == gold_doc else 0 for i in range(len(ranked))]
        precisions = [sum(relevant[:i+1]) / (i+1) for i in range(len(relevant))]
        ap = sum(p * r for p, r in zip(precisions, relevant)) / max(sum(relevant), 1)
        aps.append(ap)
    ap_at_50 = sum(aps) / len(aps)
    return acc, recall_at_k, ap_at_50

Common pitfalls

• Zero-shot evaluation prohibits fine-tuning on target benchmarks, making results highly sensitive to pre-training data overlap and domain shift.
• Retrieval performance is asymmetric; I2T and T2I Recall@10 are reported separately and can diverge significantly due to modality mismatch.
• Averaging accuracy across 10 heterogeneous benchmarks (birds, plants, fungi, camera traps) may obscure domain-specific failures or dataset scale imbalances.

Evidence (verbatim from paper)

We evaluate the natural language understanding on INQUIRE-Rerank (AP@50), Cornell Bird and PlantID(Recall@10) in [Table 2].

Citation

@misc{zhang2025biocap,
  title={BioCAP: Exploiting Synthetic Captions Beyond Labels in Biological Foundation Models},
  author={Ziheng Zhang et al.},
  year={2025},
  note={arXiv:2510.20095}
}

• arXiv: 2510.20095