Pas Dataset Eval

pas-dataset-eval

Precision at Scale: Domain-Specific Datasets On-Demand — Jesús M Rodríguez-de-Vera et al. (2024) (arXiv:2407.03463, 2024)

What this evaluates

Evaluates the transferability and pretraining quality of vision models trained on synthetic domain-specific datasets compared to manually curated and general-domain datasets. It probes the model's ability to generalize to fine-grained classification and object detection tasks within specific domains like birds and food.

Datasets

• CUB-200-2011 — total ?; splits: test (-1)
• NABirds — total ?; splits: test (-1)
• iNatbirds — total ?; splits: test (-1)
• Food-101 — total ?; splits: test (-1)
• FoodX-251 — total ?; splits: test (-1)
• Food-2K — total ?; splits: test (-1)

Metrics

• Top-1 k-NN accuracy (primary) — range: percent
  • Accuracy of a k-nearest neighbors classifier on frozen backbone features. Predicts the majority class among the k closest training examples in feature space.
• Linear accuracy — range: percent
  • Accuracy of a linear probe trained on frozen backbone features. A linear classifier is trained on the extracted features and evaluated on the test set.
• mAP — range: percent
  • Mean Average Precision for object detection tasks, measuring the precision-recall curve across confidence thresholds.

Input / output format

Input: RGB images from the target domain (birds or food) along with their ground-truth class labels or bounding boxes.

Output: Class predictions (for classification) or bounding box coordinates and class scores (for object detection).

Scoring recipe

def evaluate_knn(features, labels, k=5):
    dists = np.linalg.norm(test_features[:, None] - features[None, :], axis=2)
    topk = np.argsort(dists, axis=1)[:, :k]
    preds = [np.bincount(labels[idx]).argmax() for idx in topk]
    return np.mean(np.array(preds) == test_labels)

def evaluate_linear(features, labels, epochs=50):
    clf = LogisticRegression(max_iter=epochs)
    clf.fit(features, labels)
    return clf.score(test_features, test_labels)

Common pitfalls

• The paper evaluates both k-NN and linear probing on frozen features, but does not specify the value of k for k-NN or the exact optimizer/learning rate for the linear probe in the main text (referenced in supplementary).
• Data leakage is mitigated by filtering images resembling test sets, but the exact similarity threshold (0.45) and duplicate removal method (SSCD) must be carefully applied to avoid inflating downstream performance.
• Pretraining uses MoCo v3 for ViTs and NNCLR for ResNets with different epoch counts (300 vs 500); mixing these setups without adhering to the specified protocols will yield non-comparable results.

Evidence (verbatim from paper)

We report Top-1 k-NN and Linear accuracies for all datasets. As can be seen in Table [1], PaS datasets prove to be better pretrainers even at the same scale.

Citation

@misc{rodriguez2024precisionatscale,
  title={Precision at Scale: Domain-Specific Datasets On-Demand},
  author={Jesús M Rodríguez-de-Vera et al. (2024)},
  year={2024},
  note={arXiv:2407.03463}
}

• arXiv: 2407.03463