Mcmc Eclipse Fitting Benchmark Eval

mcmc-eclipse-fitting-benchmark-eval

Benchmark Tests for Markov Chain Monte Carlo Fitting of Exoplanet Eclipse Observations — Rogers et al. (2013) (arXiv:1302.6607, 2013)

What this evaluates

This benchmark evaluates the accuracy and reliability of Markov Chain Monte Carlo (MCMC) light curve fitting routines for detecting and measuring exoplanet secondary eclipses. It probes how well analysis pipelines recover known eclipse depths and phase centers under varying noise conditions, including synthetic white/red noise and real observational systematics.

Datasets

• MCMC Eclipse Benchmark Suite — total 10; splits: synthetic (7), real-noise-injected (3); repo http://www.pha.jhu.edu/~rogers/vhj/mcmc/benchmark/

Metrics

• eclipse depth (primary) — range: ppm
  • Absolute difference between the MCMC-recovered eclipse depth and the known input depth (in ppm).
• mid-eclipse phase — range: phase units
  • Absolute difference between the recovered mid-eclipse phase and the known input phase.

Input / output format

Input: Three-column ASCII files containing orbital phase, observed flux, and flux uncertainty per observation. Real-noise datasets additionally include columns for systematic trend variables (offset, z, w, x, y, s).

Output: Recovered eclipse depth (ppm) and mid-eclipse phase from the MCMC fitting routine.

Scoring recipe

def evaluate(predictions, gold):
    depth_err = abs(predictions['recovered_depth'] - gold['input_depth'])
    phase_err = abs(predictions['recovered_phase'] - gold['input_phase'])
    return {'depth_error_ppm': depth_err, 'phase_error': phase_err}

Common pitfalls

• Synthetic white-noise models may overestimate routine accuracy, as red-noise and real systematics introduce systematic biases and directional errors.
• Complex trend modeling on real data can still yield false positive eclipse signals in non-Gaussian noise distributions.
• Failing to test on datasets with injected signals (like R1-R3) misses the ability to verify that a routine correctly isolates the added signal from intrinsic stellar variability.

Evidence (verbatim from paper)

The depth and central phase of the eclipses that are input are provided, so that any team can check the results from their analysis routine against them.

Citation

@misc{rogers2013mcmceclipsebenchmark,
  title={Benchmark Tests for Markov Chain Monte Carlo Fitting of Exoplanet Eclipse Observations},
  author={Rogers et al. (2013)},
  year={2013},
  note={arXiv:1302.6607}
}

• arXiv: 1302.6607