Dark Machines Anomaly Score Eval

dark-machines-anomaly-score-eval

The Dark Machines Anomaly Score Challenge: Benchmark Data and Model Independent Event Classification for the Large Hadron Collider — Aarrestad et al. (2021) (arXiv:2105.14027, 2021)

What this evaluates

Evaluates the ability of unsupervised machine learning models to detect deviations from Standard Model physics in high-energy collider data without assuming specific new physics signatures. It probes model-agnostic anomaly detection by measuring how well density estimation and reconstruction-based methods separate background events from potential signal events.

Datasets

• Dark Machines Anomaly Score Challenge Dataset — total 1000000000; splits: train (-1), validation (-1), test (-1); repo https://github.com/bostdiek/DarkMachines-UnsupervisedChallenge

Metrics

• reconstruction loss (primary) — range: other
  • Computed as the mean squared error (MSE) or Chamfer distance between the input event features and the model's reconstructed output. Optionally combined with a KL divergence term to regularize the latent space. Higher values indicate greater deviation from the background distribution.

Input / output format

Input: Variable-length set of particle 4-vectors ($p_{\mathrm{T}},\eta,\phi,E$) and missing transverse energy, padded to a fixed maximum number of objects (e.g., 8 or 20). Features are standardized using training set statistics.

Output: Scalar anomaly score per event.

Scoring recipe

def compute_anomaly_score(events, model, beta=0.5):
    recon = model(events)
    # Compute reconstruction error (MSE or Chamfer)
    loss = mean_squared_error(events, recon)
    # Add KL divergence term if applicable
    kl = compute_kl_divergence(model.latent_dist)
    score = (1 - beta) * loss + beta * kl
    return score

Common pitfalls

• Failing to respect permutation invariance when processing variable-length particle sets, leading to inconsistent scores across shuffled inputs.
• Standardizing features using statistics from the entire dataset instead of only the training set, causing data leakage and artificially inflated anomaly scores.
• Arbitrarily selecting thresholds on continuous anomaly scores without calibrating against background-only validation data.

Evidence (verbatim from paper)

Not only the reconstruction loss can be a good outlier variable. Because the KL-divergence favors inputs that are encoded near the center of the latent space, the radius from the center is another anomaly score definition that can have predictive results.

Citation

@misc{aarrestad2021darkmachines,
  title={The Dark Machines Anomaly Score Challenge: Benchmark Data and Model Independent Event Classification for the Large Hadron Collider},
  author={Aarrestad et al. (2021)},
  year={2021},
  note={arXiv:2105.14027}
}

• arXiv: 2105.14027