Exoplanet Detection Eval

exoplanet-detection-eval

NA-SODINN: a deep learning algorithm for exoplanet image detection based on residual noise regimes — Cantero et al. (2023) (arXiv:2302.02854, 2023)

What this evaluates

Evaluates high-contrast imaging algorithms' ability to detect injected exoplanet companions in real and simulated ADI sequences. It measures detection sensitivity and specificity across different noise regimes and instrument datasets, comparing performance against standard PCA and deep learning baselines.

Datasets

• EIDC (Exoplanet Imaging Data Challenge) Phase 1 — total 9; splits: test (9)

Metrics

• F1-score (primary) — range: [0, 1]
  • F1 = 2TP / (2TP + FP + FN). Balances precision and recall for exoplanet detection.
• AUC_TPR — range: [0, 1]
  • Area under the True Positive Rate curve across a range of detection thresholds. Aims to be as close to 1 as possible.
• AUC_FPR — range: [0, 1]
  • Area under the False Positive Rate curve across thresholds. Aims to be as close to 0 as possible.
• AUC_FDR — range: [0, 1]
  • Area under the False Discovery Rate curve across thresholds. Aims to be as close to 0 as possible.

Input / output format

Input: Pre-processed temporal cubes of ADI images (parallactic angles corrected, with non-coronagraphic PSF and pixel-scale metadata), optionally containing injected synthetic companions at known positions and fluxes.

Output: Detection maps or confidence maps per ADI sequence, where pixel values represent the algorithm's confidence in a companion's presence.

Scoring recipe

for each threshold in threshold_range:
  TP = count(detections within FWHM aperture of injection & above threshold)
  FP = count(detections in FOV & above threshold) - TP
  FN = count(injected companions not detected)
  TN = total_pixels - TP - FP - FN
  TPR = TP / (TP + FN)
  FPR = FP / (FP + TN)
  FDR = FP / (FP + TP)
  F1 = 2 * TP / (2 * TP + FP + FN)
# ROC uses TPR vs mean_FPs_per_FOV
# AUC_TPR, AUC_FPR, AUC_FDR computed by integrating over thresholds

Common pitfalls

• The paper replaces standard FPR with 'mean number of FPs within the whole field of view' for ROC curves to better suit HCI tasks.
• Models are trained once per ADI sequence rather than retrained for each injected companion, which may slightly perturb the negative class but saves computation.
• Detection is defined strictly by a blob with at least one pixel above threshold inside a circular aperture of diameter FWHM centered on the injection.

Evidence (verbatim from paper)

The following standard metrics are then used to assess the detection performance of each submitted detection map: True Positive Rate: TPR=TP/(TP+FN), False Positive Rate: FPR=FP/(FP+TN), False Discovery Rate: FDR=FP/(FP+TP), F1-score: F1=2TP/(2TP+FP+FN).

Citation

@misc{cantero2023nasodinn,
  title={NA-SODINN: a deep learning algorithm for exoplanet image detection based on residual noise regimes},
  author={Cantero et al. (2023)},
  year={2023},
  note={arXiv:2302.02854}
}

• arXiv: 2302.02854