Gnn Vs Dnn Lhc Bb Eval

gnn-vs-dnn-lhc-bb-eval

A case study of sending graph neural networks back to the test bench for applications in high-energy particle physics — Emanuel Pfeffer et al. (arXiv:2402.17386, 2024)

What this evaluates

Compares graph neural networks against deep fully-connected feedforward networks for binary classification of b-quark pairs in top-quark-antiquark collisions at the LHC. It probes whether explicit relational inductive biases in GNNs outperform permutation-invariant DNNs when provided with equivalent kinematic and relational features.

Datasets

• LHC t\bar{t} b\bar{b} event classification — total ?; splits: train (-1)

Metrics

• mean ROC-AUC ($\mu_{\mathrm{AUC}}$) (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve. Computed as the average ROC-AUC over an ensemble of 10 independent training repetitions. Uncertainty $\Delta \mu_{\mathrm{AUC}}$ is derived from the sample variance of the ensemble.

Input / output format

Input: Kinematic feature vectors for reconstructed particles/jets (e.g., $\phi$, $\eta$, energy). For GNNs, these are node features on a fully connected graph with edge weights $\omega_{ij}$ encoding relational information (e.g., $\Delta R$, $\Delta R^{-1}$, or invariant mass $m$). DNNs receive flattened feature vectors $\mathbf{x}^{\mathrm{DNN}}$ or reduced versions $\mathbf{x}_{\mathrm{red}}^{\mathrm{DNN}}$.

Output: Binary classification probability scores for signal vs. background events, used to compute ROC-AUC.

Scoring recipe

auc_scores = []
for _ in range(10):
    model = train_model(data, architecture)
    auc_scores.append(compute_roc_auc(model.predictions, gold_labels))
mean_auc = np.mean(auc_scores)
uncertainty = np.std(auc_scores)
# Outlier removal: remove runs where auc > mean + 1.5*std if it changes std by >= 0.0025

Common pitfalls

• GNNs perform worse than DNNs if edge weights are random or unweighted; domain-knowledge-driven relational features are required to unlock GNN advantages.
• Performance is evaluated via ensemble averaging (10 runs) rather than a single train/test split, making variance $\Delta \mu_{\mathrm{AUC}}$ a critical reporting metric.
• DNNs implicitly capture relational information through positional encoding of parton associations, which can narrow the performance gap with GNNs.

Evidence (verbatim from paper)

The metric by which to judge the success of an NN to fulfill the task is chosen to be the mean of the ROC-AUC $\mu_{\mathrm{AUC}}$ based on the training setup, as described in Section 2.4. The bars associated with the points indicate the uncertainty $\Delta \mu_{\mathrm{AUC}}$ in $\mu_{\mathrm{AUC}}$ due to random choices in the training, as described in Section 2.4.

Citation

@misc{pfeffer2024gnntestbench,
  title={A case study of sending graph neural networks back to the test bench for applications in high-energy particle physics},
  author={Emanuel Pfeffer et al.},
  year={2024},
  note={arXiv:2402.17386}
}

• arXiv: 2402.17386