Phase No Eval

phase-no-eval

Phase Neural Operator for Multi-Station Picking of Seismic Arrivals — Sun et al. (2023) (arXiv:2305.03269, 2023)

What this evaluates

Evaluates the ability of neural phase pickers to detect P- and S-wave arrivals in continuous seismic waveforms across multi-station networks. It probes detection accuracy, timing precision, and generalization to out-of-distribution earthquake sequences under varying signal-to-noise conditions.

Datasets

• NCEDC 2020 Test Set — total 43700; splits: test (43700)
• 2019 Ridgecrest Sequence — total ?; splits: test (-1)

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall, calculated by comparing predicted arrival times against manual ground truth picks.
• Mean absolute error (MAE) of time residuals — range: seconds
  • Average absolute difference between predicted and manual arrival times in seconds.
• Standard deviation of time residuals — range: seconds
  • Standard deviation of the absolute time differences between predicted and manual picks in seconds.
• Event match rate — range: [0, 1]
  • Percentage of ground-truth catalog events successfully matched by predicted events within a 3-second time window.

Input / output format

Input: 30-second (or 60-second for baselines) seismic waveform windows sampled at 100 Hz, with multi-station spatial context provided via graph nodes. Picks are centered in the middle 30 seconds of the window.

Output: Probability distribution over time for P- and S-phase arrivals. Final picks are extracted as peaks exceeding a fixed threshold.

Scoring recipe

picks = [t for t, prob in enumerate(predictions) if prob >= threshold]
tp = len(set(picks) & set(gold))
fp = len(set(picks) - set(gold))
fn = len(set(gold) - set(picks))
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
residuals = [abs(p - g) for p, g in zip(sorted(picks), sorted(gold))]
mae = sum(residuals) / len(residuals)
std = (sum((r - mae)**2 for r in residuals) / len(residuals)) ** 0.5
matched = sum(1 for e_pred in pred_events if any(abs(e_pred - e_gold) <= 3.0 for e_gold in gold_events))
match_rate = matched / len(gold_events)

Common pitfalls

• Models are evaluated at individually optimized F1 thresholds rather than a fixed deployment threshold, which may overstate real-world performance.
• Slightly higher residual standard deviation for PhaseNO is attributed to harder, low-SNR detections rather than model failure, which can mislead readers expecting strictly lower variance.
• Catalog comparisons mix different association algorithms (GaMMA vs. SCSN/template matching) and station counts, making direct event-count comparisons unfair.

Evidence (verbatim from paper)

Our method results in the highest F1 scores for both P- and S-waves, being 0.99 and 0.98 respectively. ... PhaseNO results in the smallest mean absolute error for both P and S phases. ... The standard deviation of the pick residuals between SCSN and PhaseNO was 0.10 s for P phases and 0.14 s for S phases ... PhaseNO catalog totaling 26,176 events matched approximately 94% events in the SCSN catalog (10,673 of 11,389) with additional events, indicating the highest recall score of PhaseNO.

Citation

@misc{sun2023phaseno,
  title={Phase Neural Operator for Multi-Station Picking of Seismic Arrivals},
  author={Sun et al. (2023)},
  year={2023},
  note={arXiv:2305.03269}
}

• arXiv: 2305.03269