Seismic Picker Eval

seismic-picker-eval

Which picker fits my data? A quantitative evaluation of deep learning based seismic pickers — Münchmeyer et al. (2021) (arXiv:2110.13671, 2021)

What this evaluates

Evaluates the performance of deep learning and classical seismic phase pickers across three tasks: event detection, phase identification, and onset time picking. It probes cross-domain transfer capabilities and robustness to varying signal-to-noise ratios and waveform characteristics.

Datasets

• LenDB — total ?; splits: test (-1)
• GEOFON — total ?; splits: test (-1)
• INSTANCE — total ?; splits: test (-1)
• SCEDC — total ?; splits: test (-1)
• STEAD — total ?; splits: test (-1)
• ETHZ — total ?; splits: test (-1)
• Iquique — total ?; splits: test (-1)
• NEIC — total ?; splits: test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic (ROC) curve. Ranges from 0 to 1, where 1 indicates perfect detection and 0.5 indicates random guessing. It averages performance across all possible decision thresholds.
• MCC — range: [-1, 1]
  • Matthews Correlation Coefficient. Measures the quality of binary/multiclass classifications. Symmetric and independent of positive/negative class choice. Ranges from -1 (total disagreement) to 1 (full agreement).
• MAE — range: seconds
  • Mean Absolute Error of onset time residuals (predicted pick time minus manual reference time). Measures average magnitude of errors without considering direction.
• RMSE — range: seconds
  • Root Mean Squared Error of onset time residuals. Sensitive to outliers, providing a measure of the standard deviation of the prediction errors.

Input / output format

Input: Seismic waveform traces (typically 3-component) provided as fixed-length input windows (e.g., 27 s, 60 s) containing background noise and seismic arrivals.

Output: Per trace: detection probability/score, predicted phase label (P, S, or noise), and predicted onset time (sample index or seconds).

Scoring recipe

# Task 1: Event Detection (AUC)
fpr, tpr, _ = roc_curve(gold_labels, pred_scores)
auc = auc(fpr, tpr)

# Task 2: Phase Identification (MCC)
mcc = matthews_corrcoef(gold_labels, pred_labels)

# Task 3: Onset Time (MAE/RMSE)
residuals = [pred_time - gold_time for pred_time, gold_time in zip(predictions, golds)]
mae = mean(abs(residuals))
rmse = sqrt(mean([r**2 for r in residuals]))

Common pitfalls

• AUC values can be dominated by the asymptotic behavior of ROC curves, potentially masking poor performance in the low false-positive rate regime required for practical deployment.
• Onset time metrics (MAE/RMSE) are highly sensitive to dataset-specific labeling artifacts, such as fixed pick positions within input windows, which can artificially inflate performance for models with large receptive fields.
• Phase identification scores (MCC) can be misleading on datasets with severe class imbalance (e.g., very few S-wave picks), as the model may appear to perform adequately on the majority class while failing on minority phases.

Evidence (verbatim from paper)

We evaluate the first task, event detection, using receiver operating characteristics (ROC) and the corresponding area under the curve (AUC). An AUC of 1 indicates a perfect model, an AUC of 0.5 a coin toss. We evaluate task 2 using the Matthews correlation coefficient (MCC). The MCC is symmetric, i.e., in contrast to the AUC or F1 score independent on a choice of positive and negative class. It takes values between -1 (total disagreement) and 1 (full agreement). For evaluating task 3, we use the residuals, i.e., the differences between ML-pick time and the manual reference pick time. We analyze the fraction of samples with high residuals ( >0.45 s for regional, >1.5 s for teleseismic), the root mean squared error (RMSE), and the mean absolute error (MAE).

Citation

@misc{munchmeyer2021picker,
  title={Which picker fits my data? A quantitative evaluation of deep learning based seismic pickers},
  author={Münchmeyer et al. (2021)},
  year={2021},
  note={arXiv:2110.13671}
}

• arXiv: 2110.13671