Significance Eval

significance-eval

Machine-Learning Performance on Higgs-Pair Production Associated with Dark Matter at the LHC — Arganda et al. (2024) (arXiv:2401.03178, 2024)

What this evaluates

Evaluates the ability of ML classifiers to distinguish signal from background events in high-energy physics simulations. It measures classification quality via AUC and quantifies discovery potential using a likelihood-ratio-based significance metric optimized over a probability threshold.

Datasets

• Simulated di-Higgs + DM benchmarks — total 14; splits: test (-1)

Metrics

• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve, measuring the classifier's overall ability to rank signal events higher than background events across all thresholds.
• significance (primary) — range: other
  • Computed as S = sqrt(-2((S+B)ln(B/(S+B))+S)), where S and B are the signal and background rates after applying a probability threshold. Optimized to maximize discovery potential, with adjustments for systematic uncertainties via Eq. (10).

Input / output format

Input: Kinematic features of simulated particle collision events (signal and background).

Output: Probability score indicating the likelihood of the event being a signal event.

Scoring recipe

1. Train XGBoost/DNN on simulated benchmarks.
2. Compute AUC on test set.
3. For each probability threshold:
   a. Calculate signal rate S and background rate B after threshold cut.
   b. If no systematics: significance = sqrt(-2 * ((S+B)*ln(B/(S+B)) + S))
   c. If systematics included: use Eq. (10) to compute significance.
4. Select threshold that maximizes significance.
5. Report max significance and corresponding S/B rates.

Common pitfalls

• Optimizing the probability threshold for accuracy or F1-score instead of the significance metric leads to suboptimal discovery potential.
• Ignoring systematic uncertainties on background rates significantly degrades significance and shifts the optimal threshold toward stricter background rejection.
• Assuming higher AUC always yields higher significance; in this benchmark, AUC > 0.9 is common, so significance is primarily driven by effective cross-section and background rejection.

Evidence (verbatim from paper)

For each benchmark, the probability threshold is chosen to maximize the significance computed as S = \sqrt{-2\left((S+B)\ln\left(\frac{B}{S+B}\right)+S\right)}, when potential systematic uncertainties are neglected, and using Eq. (10) otherwise. In the above expression, S and B are the signal and background rates after applying the cuts in Eq. (6) and the additional cut on the output of the classifiers.

Citation

@misc{arganda2024mlhiggsdm,
  title={Machine-Learning Performance on Higgs-Pair Production Associated with Dark Matter at the LHC},
  author={Arganda et al. (2024)},
  year={2024},
  note={arXiv:2401.03178}
}

• arXiv: 2401.03178