Trackml Edge Probability Eval

trackml-edge-probability-eval

Benchmarking GPU and TPU Performance with Graph Neural Networks — Ju et al. (2022) (arXiv:2210.12247, 2022)

What this evaluates

Probes a GNN's capability to perform edge scoring on highly sparse, irregular scientific graphs by predicting the probability that a directional connection between two 3D space-point measurements originates from the same particle.

Datasets

• TrackML — total ?; splits: test (-1)

Metrics

• edge probability (primary) — range: [0, 1]
  • Not explicitly defined in the provided section. The task requires predicting a probability score for each edge indicating whether it connects two hits from the same particle. Standard binary classification metrics (e.g., accuracy, AUC) are implied but not specified.

Input / output format

Input: Variable-sized sparse graphs per collision event (avg. 50k nodes, 250k edges). Node attributes are 3D hit positions in cylindrical coordinates. Edges represent directional connections between hits.

Output: A continuous probability score for each edge, indicating the likelihood that the edge is true (connects two hits from the same particle).

Scoring recipe

def compute_edge_probability(gold_edges, pred_scores, threshold=0.5):
    correct = sum(1 for e in gold_edges if (pred_scores[e] >= threshold) == True)
    return correct / len(gold_edges) if gold_edges else 0.0

Common pitfalls

• Graph sizes and sparsity vary per event, requiring dynamic batching or padding.
• The dataset is highly imbalanced (~20% true edges), so accuracy alone may be misleading without precision/recall or AUC.
• Hardware benchmarking in this paper prioritizes memory bandwidth and compute saturation over pure model accuracy.

Evidence (verbatim from paper)

The objective of graph neural networks is to assign a score to each edge so as to indicate the probability that the edge is true, i.e. that it connects two hits from the same particle. The graph size varies for each collision event, as shown in Figure 1. On average, there are about 50,000 nodes and 250,000 edges. Out of the 250,000 edges, the expected number of true edges is about 50,000.

Citation

@misc{ju2022benchmarking,
  title={Benchmarking GPU and TPU Performance with Graph Neural Networks},
  author={Ju et al. (2022)},
  year={2022},
  note={arXiv:2210.12247}
}

• arXiv: 2210.12247