Loff Ta Image Classification Eval

loff-ta-image-classification-eval

Learning from Offline Foundation Features with Tensor Augmentations — Konuk et al. (2024) (arXiv:2410.02527, 2024)

What this evaluates

Evaluates the classification performance of a parameter-efficient model trained on cached foundation model features with tensor augmentations. It probes the model's ability to generalize across diverse image domains, object categories, and input resolutions using only lightweight classifier heads.

Datasets

• APTOS2019 — total ?; splits: train (-1), val (-1), test (-1)
• DDSM — total ?; splits: train (-1), val (-1), test (-1)
• ISIC — total ?; splits: train (-1), val (-1), test (-1)
• AID — total ?; splits: train (-1), val (-1), test (-1)
• NABirds — total ?; splits: train (-1), val (-1), test (-1)
• Flowers102 — total ?; splits: train (-1), val (-1), test (-1)
• StanfordCars — total ?; splits: train (-1), val (-1), test (-1)
• StanfordDogs — total ?; splits: train (-1), val (-1), test (-1)
• Oxford-III Pet — total ?; splits: train (-1), val (-1), test (-1)
• Caltech-101 — total ?; splits: train (-1), val (-1), test (-1)
• SUN397 — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Top-1 accuracy: the proportion of correctly predicted class labels out of the total number of instances. The paper notes that dataset-specific metrics are used where appropriate, but accuracy is the standard for these benchmarks.

Input / output format

Input: Cached feature embeddings from foundation models (DINOv2 or CLIP), derived from input images resized to 512x512 or 256x256 depending on the dataset.

Output: Class label prediction for image classification.

Scoring recipe

correct = 0
total = len(gold_labels)
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
return correct / total

Common pitfalls

• The paper uses official splits when available, but falls back to Kornblith et al. (2019) splits otherwise; failing to replicate this fallback will cause data leakage or mismatched evaluation.
• High-resolution datasets are explicitly resized to 512x512 before feature extraction, which may alter performance compared to native-resolution evaluation.
• The paper states 'metrics appropriate to its specific evaluation criteria' are reported, so assuming a single metric across all 11 datasets without checking dataset-specific standards may lead to incorrect reporting.

Evidence (verbatim from paper)

For each dataset, we report metrics appropriate to its specific evaluation criteria. We adhere to official train/validation/test splits when available, or follow [kornblith2019better] in their absence.

Citation

@misc{konuk2024loffta,
  title={Learning from Offline Foundation Features with Tensor Augmentations},
  author={Konuk et al. (2024)},
  year={2024},
  note={arXiv:2410.02527}
}

• arXiv: 2410.02527