qhjqhj00 radio-source-classification-eval
Self-supervised learning for radio-astronomy source classification: a benchmark — Cecconello et al. (2024) (arXiv:2411.14078, 2024)
What this evaluates
Evaluates the transferability of self-supervised learning representations for classifying radio astronomy sources from interferometric cutout images. It benchmarks multiple SSL pretraining methods against ImageNet baselines using linear probing and full fine-tuning on downstream classification tasks.
Datasets
- MiraBest — total ?; splits: test (-1)
- RGZ — total ?; splits: test (-1)
- MSRS — total ?; splits: test (-1)
- VLASS — total ?; splits: test (-1)
Metrics
classification accuracy(primary) — range: percent- Accuracy = (number of correctly classified images) / (total number of images). Reported as mean ± standard deviation over 3-fold cross-validation or 3 independent random seeds.
Input / output format
Input: Radio astronomy cutout images normalized using per-image minimum and maximum pixel values, then resized to 224×224 pixels.
Output: Discrete class label indicating the radio source type.
Scoring recipe
def compute_accuracy(predictions, gold_labels):
correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
return correct / len(gold_labels)
# Report mean and std of accuracy over 3-fold CV or 3 runs
Common pitfalls
- Confusing the pretraining datasets (Curated/Uncurated) with the downstream evaluation datasets (MiraBest, RGZ, MSRS, VLASS).
- Assuming ImageNet pretraining is competitive; the benchmark shows it generally underperforms SSL on radio data except on MSRS.
- Ignoring the reporting of standard deviation; results must include mean ± std over 3 runs or 3-fold CV.
Evidence (verbatim from paper)
Evaluation results on the downstream tasks are reported in terms of classification accuracy. Using the above final hyperparameters, we train each SSL method with each backbone on each pretraining dataset for three times with random initialization. The only exception is that, for the Uncurated dataset, we only use ResNet-18, for timing constraints. Then, we evaluate each trained model on all downstream tasks, using 3-fold cross-validation on each task. In practice, for a given combination of SSL method, backbone, pretraining dataset and downstream dataset, we have nine values of accuracy, for which we report the corresponding mean and standard deviation.
Citation
@misc{cecconello2024selfsupervised,
title={Self-supervised learning for radio-astronomy source classification: a benchmark},
author={Cecconello et al. (2024)},
year={2024},
note={arXiv:2411.14078}
}
- arXiv: 2411.14078