Vit Zero Shot Clustering Eval

vit-zero-shot-clustering-eval

Vision Transformers for Zero-Shot Clustering of Animal Images: A Comparative Benchmarking Study — Hugo Markoff, Stefan Hein Bengtson, Michael Ørsted (2026) (arXiv:2602.03894, 2026)

What this evaluates

Evaluates the ability of Vision Transformer models combined with dimensionality reduction and clustering algorithms to perform zero-shot species-level clustering of animal images. It probes how well unsupervised pipelines can recover ground-truth taxonomic labels and capture intra-specific variation without manual annotation.

Datasets

• Animal Images (Birds & Mammals) — total ?; splits: test (-1)

Metrics

• V-measure (primary) — range: [0, 1]
  • Harmonic mean of clustering homogeneity and completeness with respect to ground-truth species labels. Values closer to 1 indicate perfect alignment between predicted clusters and true species.
• Isolation Index (II) — range: [0, 1]
  • Measures average purity per species: II_s = (1/N_s) * sum_{c in C} (n_{s,c}^2 / |c|), where N_s is total images of species s, n_{s,c} is count of species s in cluster c, and |c| is cluster size.
• Effective Cluster Count (ECC) — range: other
  • Measures how many clusters a species effectively 'owns': ECC_s = sum_{c in C} (n_{s,c} / |c|). Values near 1 indicate a species forms a single pure cluster.

Input / output format

Input: Unlabeled animal images (RGB, IR, or flash) representing multiple species.

Output: Discrete cluster labels assigned to each image.

Scoring recipe

def compute_vmeasure(predictions, ground_truth):
    return v_measure_score(ground_truth, predictions)

def compute_ii_ecc(predictions, ground_truth):
    clusters = defaultdict(list)
    for img_id, cluster_id in enumerate(predictions):
        clusters[cluster_id].append(ground_truth[img_id])
    ii_scores, ecc_scores = [], []
    for species in set(ground_truth):
        n_s = sum(1 for g in ground_truth if g == species)
        ii_s = sum((sum(1 for g in c if g == species)**2) / len(c) for c in clusters.values()) / n_s
        ecc_s = sum(sum(1 for g in c if g == species) / len(c) for c in clusters.values())
        ii_scores.append(ii_s)
        ecc_scores.append(ecc_s)
    return mean(ii_scores), mean(ecc_scores)

Common pitfalls

• Assuming the number of predicted clusters equals the true number of species; unsupervised methods like DBSCAN heavily over-split, while HDBSCAN slightly over-estimates.
• Interpreting cluster oversplitting as failure; intra-specific variation (age, sex, lighting) naturally fragments species into multiple pure clusters, which does not compromise identification accuracy.
• Applying supervised clustering parameters (e.g., fixed K=30) in zero-shot scenarios where the true species count is unknown.

Evidence (verbatim from paper)

DINOv3 achieves the highest average V-measure (0.817), followed by DINOv2 (0.769). In contrast, biology-specific BioCLIP 2 and general-purpose CLIP and SigLIP show substantially lower performance, with average V-measures ranging from 0.597 to 0.652.

Citation

@misc{markoff2026vitclustering,
  title={Vision Transformers for Zero-Shot Clustering of Animal Images: A Comparative Benchmarking Study},
  author={Hugo Markoff, Stefan Hein Bengtson, Michael Ørsted (2026)},
  year={2026},
  note={arXiv:2602.03894}
}

• arXiv: 2602.03894