Seismic Phase Association Eval

seismic-phase-association-eval

Benchmarking seismic phase associators: Insights from synthetic scenarios — Puente et al. (2025) (arXiv:2501.03621, 2025)

What this evaluates

Evaluates the accuracy and computational efficiency of seismic phase associators on synthetic crustal and subduction zone datasets under varying event densities and noise levels. It probes the models' ability to correctly group seismic picks into events and maintain performance under high-stress conditions.

Datasets

• Synthetic Seismic Scenarios (Crustal & Subduction) — total ?; splits: crustal (-1), subduction (-1); repo https://github.com/imcbrearty/GENIE

Metrics

• event-level F1 score (primary) — range: [0, 1]
  • Precision = TP/(TP+FP), Recall = TP/(TP+FN), F1 = 2PR/(P+R). An event is considered correctly identified (TP) if the predicted event shares at least 50% of its picks with the ground truth event.
• pick-level F1 score — range: [0, 1]
  • Calculated on correctly associated (CA), wrongly associated (WAP), and false picks. Precision = CA/(CA+WAP), Recall = CA/(CA+False). Only computed for events meeting the 50% match threshold.
• processing runtime — range: seconds
  • Total wall-clock time in seconds to process all picks for a given scenario run.

Input / output format

Input: Synthetic seismic datasets containing ground truth picks, noise picks, and event-station configurations for crustal or subduction zones, with event counts of 100, 500, or 2000 per 24 hours and noise proportions of 30%, 100%, or 300%.

Output: Grouped lists of associated picks representing predicted seismic events, along with processing time in seconds.

Scoring recipe

def compute_event_f1(gt_events, pred_events):
    tp, fp, fn = 0, 0, 0
    for gt in gt_events:
        best_overlap = max(len(gt & pred) for pred in pred_events)
        if best_overlap >= 0.5 * len(gt):
            tp += 1
        else:
            fn += 1
    for pred in pred_events:
        if not any(len(gt & pred) >= 0.5 * len(gt) for gt in gt_events):
            fp += 1
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• Pick-level metrics are only calculated on events that exceed the 50% matching threshold, meaning false events created from unmatched picks or noise are excluded from the calculation.
• GaMMA fails to complete processing on high-complexity runs (2000 events, 300% noise) due to memory allocation issues, resulting in missing metrics for those conditions.
• Event-level precision is generally higher than recall across most associators, indicating a tendency to create fewer false events but miss more true events.

Evidence (verbatim from paper)

At low noise levels (30% noise) and small event counts (100 events), all associators demonstrate high event-level precision and recall, with relatively minor differences between different scenarios and associators. While GENIE, PyOcto and REAL show values above 0.97 for precision, recall and F1 score in both scenarios, GaMMA obtains lower scores around 0.9 for the subduction zone scenario, and PhaseLink scores around or even below 0.9 for both scenarios.

Citation

@misc{puente2025benchmarking,
  title={Benchmarking seismic phase associators: Insights from synthetic scenarios},
  author={Puente et al. (2025)},
  year={2025},
  note={arXiv:2501.03621}
}

• arXiv: 2501.03621