Slcp C2st Eval

slcp-c2st-eval

LtU-ILI: An All-in-One Framework for Implicit Inference in Astrophysics and Cosmology — Ho et al. (2024) (arXiv:2402.05137, 2024)

What this evaluates

Evaluates the accuracy of simulation-based inference engines in recovering the true posterior distribution of model parameters given synthetic observational data. It probes the ability of implicit likelihood methods to handle complex, multimodal posteriors and varying simulation budgets.

Datasets

• SLCP (Simple Likelihood Complex Posterior) — total ?; splits: test (-1)

Metrics

• C2ST (primary) — range: [0, 1]
  • Trains a binary classifier to distinguish between samples drawn from the reference (ground truth) posterior and samples from the inferred posterior. The metric equals the classification accuracy. A score of 0.5 indicates perfect recovery (indistinguishable distributions), while higher values indicate worse performance.

Input / output format

Input: Synthetic data vector x generated from a known simulator, along with the true parameters theta used to generate it.

Output: Samples or a probability density function representing the inferred posterior distribution p(theta|x).

Scoring recipe

# reference_samples: samples from ground truth posterior (e.g., via HMC)
# inferred_samples: samples from the trained inference engine
# Train a binary classifier to predict label 1 for reference_samples and 0 for inferred_samples.
# C2ST = accuracy of classifier on held-out test set of both sample types.
# Optimal score is 0.5. Lower is better.

Common pitfalls

• C2ST requires a high-quality reference posterior (typically from long-run MCMC/HMC), which can be computationally prohibitive for complex simulators.
• The metric is sensitive to the number of samples used to train the classifier and evaluate; insufficient samples can artificially inflate or deflate scores.
• Sequential methods may show negligible gains at small simulation budgets (10^3) but massive improvements at larger budgets (10^5), making budget selection critical for fair comparison.

Evidence (verbatim from paper)

Error is defined in terms of the Classifier 2-sample Test (C2ST; Lopez-Paz and Oquab, 2016) metric. A lower C2ST indicates a more accurate inferred posterior, with 0.5 being the optimal score. C2ST values are shown at their median and central 95% confidence interval, calculated over ten independent runs.

Citation

@misc{ho2024ltuili,
  title={LtU-ILI: An All-in-One Framework for Implicit Inference in Astrophysics and Cosmology},
  author={Ho et al. (2024)},
  year={2024},
  note={arXiv:2402.05137}
}

• arXiv: 2402.05137