Polaris Eval

polaris-eval

POLARIS: A High-contrast Polarimetric Imaging Benchmark Dataset for Exoplanetary Disk Representation Learning — Cao et al. (2025) (arXiv:2506.03511, 2025)

What this evaluates

Evaluates the ability of machine learning models to distinguish between reference stars and circumstellar exoplanetary disks in high-contrast polarimetric imaging data. It probes representation learning quality through downstream supervised classification and unsupervised clustering tasks.

Datasets

• POLARIS — total 909; splits: labeled (96), unlabeled (813)

Metrics

• accuracy (primary) — range: [0, 1]
  • Mean classification accuracy across 10-fold stratified cross-validation folds. For unsupervised clustering, cluster labels are aligned to ground truth using the Hungarian algorithm before computing accuracy.

Input / output format

Input: 256×256 grayscale polarimetric images. For vision-language models, images are paired with a zero-shot prompt template instructing the model to classify the image as either a target disk or reference star.

Output: Binary classification label: 'target' or 'reference'.

Scoring recipe

accuracies = []
for train_idx, test_idx in StratifiedKFold(n_splits=10).split(X, y):
    model.fit(X[train_idx], y[train_idx])
    preds = model.predict(X[test_idx])
    accuracies.append(accuracy_score(y[test_idx], preds))
# For clustering: align cluster labels to ground truth via Hungarian algorithm first
final_accuracy = mean(accuracies)

Common pitfalls

• Small labeled set (96 images) makes regression prone to overfitting; only classification/clustering is evaluated.
• Unsupervised clustering requires label alignment via the Hungarian algorithm before computing accuracy.
• Hyperparameters are tuned independently within each CV fold to prevent data leakage.

Evidence (verbatim from paper)

A 10-fold Stratified Cross-Validation (CV) procedure is applied, where hyperparameters are fine-tuned within each fold using a 5-fold grid search. The classifier is trained on the training data of each fold and evaluated on the test data. The final performance is reported as the mean accuracy across all folds. ... Cluster labels are aligned to ground truth using the Hungarian algorithm for optimal matching. All evaluations are conducted with 10-fold CV and a fixed random seed, and we report the mean accuracy across folds.

Citation

@misc{cao2025polaris,
  title={POLARIS: A High-contrast Polarimetric Imaging Benchmark Dataset for Exoplanetary Disk Representation Learning},
  author={Cao et al. (2025)},
  year={2025},
  note={arXiv:2506.03511}
}

• arXiv: 2506.03511