Earthquakenpp Eval

earthquakenpp-eval

EarthquakeNPP: A Benchmark for Earthquake Forecasting with Neural Point Processes — Stockman, Lawson, and Werner (2024) (arXiv:2410.08226, 2024)

What this evaluates

This benchmark evaluates the forecasting capability of neural spatio-temporal point processes (NPPs) on earthquake sequences. It probes how well models capture the joint temporal and spatial intensity of seismic events compared to traditional seismological baselines like ETAS.

Datasets

• EarthquakeNPP (ComCat, QTM_SaltonSea, QTM_SanJac, White, SCEDC) — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• temporal log-likelihood (primary) — range: other
  • Sum of the log of the predicted temporal intensity function evaluated at each observed event time in the test set. Higher values indicate better temporal forecasting performance.
• spatial log-likelihood — range: other
  • Sum of the log of the predicted spatial probability density function evaluated at each observed event location in the test set. Higher values indicate better spatial forecasting performance.

Input / output format

Input: Historical spatio-temporal earthquake event sequences (timestamps and coordinates) up to a defined split point, used to train or condition the model.

Output: Predicted conditional intensity function (or spatial density) over the test period, evaluated as log-likelihood scores per event.

Scoring recipe

def compute_log_likelihood(test_events, model):
    total_ll = 0.0
    for event in test_events:
        t, x = event.time, event.location
        intensity = model.predict_intensity(t, x, history)
        total_ll += np.log(intensity)
    return total_ll

Common pitfalls

• The ETAS baseline incorporates earthquake magnitude dependence in its intensity function, which is explicitly excluded from the NPP implementations in this benchmark.
• Validation data is combined with training data for fitting ETAS, whereas NPPs follow a strict train/val/test split, creating an asymmetry in model calibration.
• NSTPP lacks memory of prior events, so its likelihood evaluation on test events does not benefit from historical context like the other NPPs or ETAS.

Evidence (verbatim from paper)

Figures 2 and 3 report the temporal and spatial log-likelihood scores of all models on the EarthquakeNPP datasets. The ETAS model achieves the highest temporal and spatial log-likelihood across all datasets, with some NPP models achieving comparable temporal performance on ComCat, QTM_SaltonSea, QTM_SanJac, and White catalogs.

Citation

@misc{stockman2024earthquakenpp,
  title={EarthquakeNPP: A Benchmark for Earthquake Forecasting with Neural Point Processes},
  author={Stockman, Lawson, and Werner (2024)},
  year={2024},
  note={arXiv:2410.08226}
}

• arXiv: 2410.08226