Smash Stpp Tpp Eval

smash-stpp-tpp-eval

Score Matching-based Pseudolikelihood Estimation of Neural Marked Spatio-Temporal Point Process with Uncertainty Quantification — Li et al. (2023) (arXiv:2310.16310, 2023)

What this evaluates

Evaluates neural marked spatio-temporal and temporal point process models on predicting the next event's time, location, and mark, while quantifying prediction uncertainty. It probes the model's ability to generate well-calibrated confidence regions for continuous variables and accurate probability estimates for discrete marks.

Datasets

• Earthquake — total ?; splits: test (-1); repo https://earthquake.usgs.gov/earthquakes/search/
• Crime — total ?; splits: test (-1); repo https://www.atlantapd.org
• Football — total ?; splits: test (-1); repo WyScout Open Access Dataset
• StackOverflow — total ?; splits: test (-1)
• Retweet — total ?; splits: test (-1)
• MIMIC-II — total ?; splits: test (-1)
• Financial Transactions — total ?; splits: test (-1)

Metrics

• Calibration Score (CS) (primary) — range: percent
  • Average calibration error across confidence levels. For each level, computes the absolute difference between the actual coverage of the generated confidence region and the desired confidence level.
• Mean Absolute Error (MAE) — range: percent
  • Mean absolute difference between the point prediction (average of generated samples) and the ground truth for event times and locations.
• Mark Prediction Accuracy (Acc) — range: percent
  • Accuracy of the predicted event mark, determined by taking the mode of the marks from the generated samples.
• Expected Calibration Error (ECE) — range: percent
  • Weighted average of the absolute difference between predicted probabilities and observed frequencies of mark outcomes across probability bins.

Input / output format

Input: Historical sequence of events (timestamps, spatial coordinates, categorical marks) observed up to a given time step.

Output: A collection of generated samples for the next event (time, location, mark), along with empirical confidence regions and mark probabilities.

Scoring recipe

def score(samples, gt):
    # MAE: average of samples vs ground truth
    mae_t = mean(abs(s.t - gt.t) for s in samples)
    mae_l = mean(dist(s.l, gt.l) for s in samples)
    
    # CS: avg calibration error over confidence levels
    cs = 0
    for alpha in confidence_levels:
        region = get_conf_region(samples, alpha)
        cs += abs(1 if gt in region else 0 - alpha)
    cs /= len(confidence_levels)
    
    # Acc: mode of sample marks
    pred_mark = mode([s.m for s in samples])
    acc = 1 if pred_mark == gt.m else 0
    
    # ECE: weighted avg of |bin_conf - bin_acc|
    ece = 0
    for bin in mark_bins:
        ece += len(bin) * abs(mean(bin.conf) - mean(bin.acc))
    ece /= len(samples)
    return cs, mae_t, mae_l, acc, ece

Common pitfalls

• Point predictions for time and location are derived by averaging samples, which can be highly misleading for multi-modal distributions.
• Confidence regions are constructed empirically from generated samples rather than analytically, making coverage sensitive to sample size and sampling method.
• Mark prediction relies on the mode of discrete samples, which may not align with the highest-probability class if the score-based sampler under-represents certain modes.

Evidence (verbatim from paper)

We employ four metrics to evaluate the quality of the generated event samples: • Calibration Score (CS) measures the uncertainty quantification performance of the samples’ time and location. It first computes confidence regions at different confidence levels from the samples, and then calculates the calibration error for each level. The calibration error is determined by the difference between the actual coverage of the region and the desired confidence level. The final metric is defined as the average calibration error across all confidence regions. A lower CS denotes superior performance in quantifying uncertainty. In our experiment, we compute the average error at confidence levels from 0.5 to 1 in increments of 0.1 for STPPs, and from 0.8 to 1 in increments of 0.05 for TPPs. This choice is motivated by that confidence intervals/regions with higher levels are typically more useful. • Mean Absolute Error (MAE) measures the mean difference between the point prediction and the ground truth of the events’ times and locations. We make point predictions by calculating the average time and location of the generated samples. • Mark Prediction Accuracy (Acc) calculates the accuracy of th

Citation

@misc{li2023smash,
  title={Score Matching-based Pseudolikelihood Estimation of Neural Marked Spatio-Temporal Point Process with Uncertainty Quantification},
  author={Li et al. (2023)},
  year={2023},
  note={arXiv:2310.16310}
}

• arXiv: 2310.16310