Cosmic Symmetry Benchmark Eval

cosmic-symmetry-benchmark-eval

A Cosmic-Scale Benchmark for Symmetry-Preserving Data Processing — Balla et al. (2024) (arXiv:2410.20516, 2024)

What this evaluates

Evaluates the ability of graph neural networks to extract cosmological parameters and local velocity fields from large-scale 3D point clouds of dark matter halos. It probes both global long-range correlation capture (via cosmological parameter regression) and local geometric dependency modeling (via per-node velocity prediction).

Datasets

• Quijote BSQ Point Cloud — total 12384; splits: train (2048), val (512), test (512); repo https://github.com/smsharma/eqnn-jax

Metrics

• MSE (primary) — range: other
  • Mean Squared Error between predicted values and ground truth targets. For graph-level tasks, averaged over the two cosmological parameters; for node-level tasks, averaged over all 5000 points and 3 velocity components.

Input / output format

Input: 3D point cloud of 5000 dark matter halo positions, represented as a tensor of shape R^{5000x3}.

Output: Graph-level: 2D vector [Omega_m, sigma_8]. Node-level: 3D velocity tensor of shape R^{5000x3}.

Scoring recipe

def compute_mse(pred, target):
    return np.mean((pred - target) ** 2)

# Graph-level: pred/target shape (2,)
# Node-level: pred/target shape (5000, 3)
graph_mse = compute_mse(pred_graph, target_graph)
node_mse = compute_mse(pred_node, target_node)

Common pitfalls

• The dataset uses only the 5000 most massive halos per simulation, not the full particle set, which alters point cloud density and correlation structure compared to raw N-body outputs.
• The node-level task predicts instantaneous velocity vectors from static positions alone, lacking explicit temporal or dynamical information, making it inherently challenging.
• Cosmological parameters Omega_m and sigma_8 are often degenerate in summary statistics, so independent regression performance can be sensitive to architectural inductive biases rather than just data fitting.

Evidence (verbatim from paper)

We consider two benchmark tasks to evaluate the performance of our models: a graph-level and a node-level prediction task. ... We train the models using the mean squared error (MSE) loss between the predicted output and true target parameters. ... We train the models using the MSE loss on the predicted velocities.

Citation

@misc{balla2024cosmic,
  title={A Cosmic-Scale Benchmark for Symmetry-Preserving Data Processing},
  author={Balla et al. (2024)},
  year={2024},
  note={arXiv:2410.20516}
}

• arXiv: 2410.20516