Backbone Optimizer Coupling Eval

backbone-optimizer-coupling-eval

Unveiling the Backbone-Optimizer Coupling Bias in Visual Representation Learning — Hippocampus et al. (2024) (arXiv:2410.06373, 2024)

What this evaluates

Probes the interdependence between vision backbone architectures and optimization algorithms by measuring how different backbones perform when paired with various optimizers across classification and detection tasks. It evaluates whether architectural design dictates optimal optimizer choice and how this coupling affects transfer learning and hyperparameter robustness.

Datasets

• CIFAR-100 — total ?; splits: train (-1), test (-1)
• ImageNet-1K — total ?; splits: train (-1), val (-1)
• COCO — total ?; splits: train (-1), val (-1)

Metrics

• Top-1 accuracy (primary) — range: [0, 1]
  • The proportion of correctly classified images out of the total number of images in the dataset. Computed as correct predictions divided by total samples.
• AP50 — range: [0, 1]
  • Average Precision for object detection and pose estimation computed at an Intersection over Union (IoU) threshold of 0.50. Measures detection/pose quality across recall levels.

Input / output format

Input: Pre-trained vision backbones (e.g., ResNet, ViT, ConvNeXt) initialized with weights from various optimizers, fed with image data from CIFAR-100, ImageNet-1K, or COCO for classification, object detection, or pose estimation tasks.

Output: Predicted class labels for classification tasks, or bounding boxes and keypoints for COCO tasks, evaluated against ground truth annotations.

Scoring recipe

# Pseudo-code for metric computation
def compute_top1_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
    return correct / len(gold_labels)

def compute_ap50(predictions, gold_annotations):
    # COCO standard evaluation at IoU=0.50
    return coco_api.accumulate(predictions, gold_annotations, iouThr=0.50)['AP']

# Apply per dataset/task
if task == 'classification':
    metric_value = compute_top1_accuracy(preds, labels)
elif task in ['detection', 'pose']:
    metric_value = compute_ap50(preds, annotations)

Common pitfalls

• Assuming a single optimizer (e.g., AdamW) is universally optimal regardless of backbone architecture, ignoring the documented coupling bias.
• Overlooking hyperparameter sensitivity; classical CNNs require narrow, specific learning rate and weight decay ranges, while modern DNNs tolerate broader ranges.
• Directly swapping optimizers during transfer learning without re-tuning, which degrades performance due to the bidirectional coupling bias.

Evidence (verbatim from paper)

Table[2] details the Top-1 accuracy for a curated selection of vision backbones under various optimizers. The results are congruent with those from CIFAR-100, reinforcing the BOCB phenomenon.

Citation

@misc{hippocampus2024backbone,
  title={Unveiling the Backbone-Optimizer Coupling Bias in Visual Representation Learning},
  author={Hippocampus et al. (2024)},
  year={2024},
  note={arXiv:2410.06373}
}

• arXiv: 2410.06373