Ibd Selection Eval

ibd-selection-eval

Interpretable machine learning approach for electron antineutrino selection in a large liquid scintillator detector — A. Gavrikov et al. (2024) (arXiv:2406.12901, 2024)

What this evaluates

Evaluates machine learning models' ability to classify electron antineutrino (IBD) events from background accidents in a liquid scintillator detector. It measures how well the models preserve signal efficiency while controlling background contamination compared to traditional cut-based selection.

Datasets

• JUNO IBD/Accident Dataset — total 152500000; splits: test (152500000)

Metrics

• efficiency (primary) — range: percent
  • Fraction of true IBD events correctly classified as IBD. Calculated as (True Positives) / (Total IBD events).
• purity — range: percent
  • Fraction of selected events that are true IBDs. Calculated as (True Positives) / (True Positives + False Positives).
• F1-score — range: [0, 1]
  • Harmonic mean of purity and efficiency: 2 * (purity * efficiency) / (purity + efficiency). Used to balance signal retention and background rejection.
• background_level — range: other
  • Fraction of selected events that are background accidents. Calculated as (False Positives) / (Total Accident events).

Input / output format

Input: Numerical detector features per event, including spatial separation ($\Delta R$), time delay ($\Delta t$), and other reconstructed kinematic variables.

Output: Continuous confidence score in [0, 1] via sigmoid function, thresholded to binary IBD/accident classification.

Scoring recipe

def compute_metrics(predictions, gold_labels, threshold=0.47):
    binary_preds = (predictions >= threshold).astype(int)
    tp = np.sum((binary_preds == 1) & (gold_labels == 1))
    fp = np.sum((binary_preds == 1) & (gold_labels == 0))
    total_ibd = np.sum(gold_labels == 1)
    total_acc = np.sum(gold_labels == 0)
    efficiency = tp / total_ibd
    purity = tp / (tp + fp)
    f1 = 2 * (purity * efficiency) / (purity + efficiency)
    background_level = fp / total_acc
    return efficiency, purity, f1, background_level

Common pitfalls

• Threshold selection is highly sensitive to the physics analysis goal; optimizing for F1-score may not align with analyses requiring maximum efficiency or strict background suppression.
• Background level is extremely low (~10^-6), necessitating massive accident samples (e.g., 150M events) for statistically robust evaluation.
• Fiducial volume cuts drastically change baseline efficiency and must be explicitly matched when comparing ML models to cut-based benchmarks.

Evidence (verbatim from paper)

Combined with the testing dataset, it consists of 152.5 million events, with 2.5 million events being IBD and the rest being accidents. ... To evaluate the performances of the FCNN and BDT models and to compare it with the cut-based selection approach, we use efficiency as the main metric with an additional condition on the background level, i.e., the fraction of selected (classified as IBDs) accidents with respect to the total number.

Citation

@misc{gavrikov2024interpretable,
  title={Interpretable machine learning approach for electron antineutrino selection in a large liquid scintillator detector},
  author={A. Gavrikov et al. (2024)},
  year={2024},
  note={arXiv:2406.12901}
}

• arXiv: 2406.12901