Patch Selectivity Eval

patch-selectivity-eval

Hardwiring ViT Patch Selectivity into CNNs using Patch Mixing — Lee et al. (2023) (arXiv:2306.17848, 2023)

What this evaluates

Evaluates a model's ability to ignore out-of-context patches (patch selectivity) and maintain classification accuracy under simulated occlusion and spatial permutation attacks.

Datasets

• ImageNet-1K val — total ?; splits: val (-1)
• SMD — total ?; splits: test (-1)
• NVD — total ?; splits: val (-1)
• ROD — total ?; splits: test (-1)

Metrics

• Top-1 accuracy (primary) — range: percent
  • Standard classification accuracy: the fraction of correctly predicted top-1 classes out of total evaluated instances.
• Top-5 accuracy — range: percent
  • Fraction of instances where the true label appears in the model's top-5 predicted classes.
• Inverse patch selectivity — range: [0, 1]
  • Sum of Softmax-normalized c-RISE importance values over out-of-context patches. Lower values indicate better selectivity.

Input / output format

Input: Natural images from benchmark datasets, optionally subjected to patch replacement, grid shuffling, or occlusion attacks during evaluation.

Output: Class predictions (top-1 or top-5 ranked classes) and, for explainability analysis, c-RISE importance heatmaps normalized via Softmax.

Scoring recipe

def compute_accuracy(preds, gold):
    correct = sum(1 for p, g in zip(preds, gold) if p == g)
    return correct / len(gold)

def compute_inverse_patch_selectivity(rise_maps, ooc_mask):
    norm_maps = softmax(rise_maps, dim=0)
    return sum(norm_maps[ooc_mask])

Common pitfalls

• Confusing the training-time Patch Mixing augmentation with the evaluation-time patch replacement/shuffle attacks.
• Mixing up Top-1 and Top-5 accuracy metrics, as the paper reports Top-1 for IN/SMD but Top-5 for NVD/ROD.
• Assuming ViTs inherently lack occlusion robustness without testing under controlled information loss.

Evidence (verbatim from paper)

Table 1 presents a summary of the results for different network architectures tested on three datasets: ImageNet-1K val (IN) top-1, SMD top-1 (avg. over 10 - 30% occlusion), NVD [25] simulated occlusion validation top-5, and ROD top-5.

Citation

@misc{lee2023hardwiring,
  title={Hardwiring ViT Patch Selectivity into CNNs using Patch Mixing},
  author={Lee et al. (2023)},
  year={2023},
  note={arXiv:2306.17848}
}

• arXiv: 2306.17848