Fewstab Eval

fewstab-eval

Benchmarking Spurious Bias in Few-Shot Image Classifiers — Zheng et al. (2024) (arXiv:2409.02882, 2024)

What this evaluates

This benchmark evaluates the robustness of few-shot image classifiers to spurious class-attribute correlations. It constructs evaluation tasks where support sets contain images with engineered spurious attributes, and query sets lack these attributes or contain attributes from other classes, then measures classification accuracy compared to standard random task sampling.

Datasets

• miniImageNet — total ?; splits: train (-1), val (-1), test (-1)
• tieredImageNet — total ?; splits: train (-1), val (-1), test (-1)
• CUB-200 — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• wAcc-A (primary) — range: percent
  • Accuracy on FewSTAB-constructed tasks where support sets are built using inter-class and intra-class attribute-based sampling to introduce spurious correlations. Computed as the percentage of correctly classified query images across 3000 evaluation tasks.
• wAcc-R — range: percent
  • Accuracy on standard randomly sampled few-shot tasks. Computed as the percentage of correctly classified query images across 3000 evaluation tasks.

Input / output format

Input: Support set images (few samples per class, resized to 84x84) and query set images (15 samples per class, resized to 84x84). Tasks are constructed either via FewSTAB's attribute-based sampling or random sampling.

Output: Class predictions for each query image in the task.

Scoring recipe

def compute_accuracy(tasks, model):
    correct = 0
    total = 0
    for task in tasks:
        preds = model.predict(task.query_images, task.support_images)
        correct += sum(p == gt for p, gt in zip(preds, task.query_labels))
        total += len(task.query_labels)
    return (correct / total) * 100

wAcc_A = compute_accuracy(fewstab_tasks, model)
wAcc_R = compute_accuracy(random_tasks, model)

Common pitfalls

• High standard accuracy (Acc) does not correlate well with robustness to spurious bias (wAcc-A); a model can perform well on random tasks but fail on FewSTAB tasks.
• The rank correlation between wAcc-A and wAcc-R is high in 1-shot settings because spurious correlations are harder to exploit with only one support sample per class, making FewSTAB less effective at revealing bias differences at 1-shot.
• FewSTAB's results are robust to the choice of VLM used for attribute detection, so switching between ViT-GPT2 and BLIP does not significantly change method rankings.

Evidence (verbatim from paper)

We show in Table [3] the wAcc (Eq. ([2])) on 5-way 1/5-shot test tasks that are randomly sampled (wAcc-R) and are constructed with FewSTAB (wAcc-A), respectively. FewSTAB generates FSC test tasks only based on the class-attribute correlations in data. In each test setting, the FSC methods in Table [3] are evaluated with the same FSC tasks.

Citation

@misc{zheng2024fewstab,
  title={Benchmarking Spurious Bias in Few-Shot Image Classifiers},
  author={Zheng et al. (2024)},
  year={2024},
  note={arXiv:2409.02882}
}

• arXiv: 2409.02882