qhjqhj00 cmc-downstream-eval
Contrastive Multiview Coding — Tian et al. (2019) (arXiv:1906.05849, 2019)
What this evaluates
Evaluates the transferability and quality of self-supervised multiview representations by measuring downstream performance on image classification, video action recognition, and semantic segmentation tasks.
Datasets
- ImageNet — total ?; splits: test (-1)
- UCF-101 — total ?; splits: test (-1)
- HMDB-51 — total ?; splits: test (-1)
- NYU-Depth-V2 — total 1449; splits: test (-1)
- STL-10 — total ?; splits: test (-1)
Metrics
Top-1 classification accuracy (%)(primary) — range: percent- Percentage of correctly predicted class labels out of the total number of test samples.
Top-5 classification accuracy (%)— range: percent- Percentage of test samples where the true label appears in the top 5 predicted classes.
Pixel Accuracy (%)— range: percent- Percentage of correctly classified pixels out of the total number of pixels in the segmentation task.
mIoU (%)— range: percent- Mean Intersection over Union across all semantic classes, computed as the average of IoU per class.
Input / output format
Input: RGB images converted to specific channel splits (e.g., L and ab in Lab space, Y and DbDr in YCbCr), video frames, optical flow maps, depth maps, surface normals, or semantic labels. For contrastive pre-training, 128x128 patches are randomly cropped from original images.
Output: Class predictions from a linear classifier or segmentation decoder trained on frozen encoder features.
Scoring recipe
def compute_accuracy(preds, targets):
correct = (preds == targets).sum()
return correct / len(targets) * 100
def compute_miou(preds, targets, num_classes):
ious = []
for c in range(num_classes):
intersection = ((preds == c) & (targets == c)).sum()
union = ((preds == c) | (targets == c)).sum()
ious.append(intersection / union if union > 0 else 0)
return sum(ious) / num_classes * 100
Common pitfalls
- Freezing the pre-trained encoder weights before training the downstream linear classifier or decoder, as required by the protocol.
- Using single-crop evaluation for ImageNet accuracy rather than multi-crop or test-time augmentation.
- Evaluating different view combinations (e.g., L vs ab vs depth) requires retraining the contrastive model from scratch, not just swapping input channels.
Evidence (verbatim from paper)
The quality of such a representation is evaluated by freezing the weights of encoder and training linear classifier on top of each layer. Table 1: Top-1 / Top-5 Single crop classification accuracy (%) on ImageNet with a supervised logistic regression classifier.
Citation
@misc{tian2019contrastive,
title={Contrastive Multiview Coding},
author={Tian et al. (2019)},
year={2019},
note={arXiv:1906.05849}
}
- arXiv: 1906.05849