Phase Picking Eval

phase-picking-eval

SeisLM: a Foundation Model for Seismic Waveforms — Tianlin Liu et al. (2024) (arXiv:2410.15765, 2024)

What this evaluates

Evaluates a model's ability to detect seismic events and classify phase types (P vs S) from raw waveform windows, particularly under varying amounts of labeled training data. It probes the effectiveness of self-supervised pretraining in learning generalizable seismic features compared to randomly initialized baselines.

Datasets

• ETHZ — total 22000; splits: train (-1), test (-1)
• GEOFON — total 161000; splits: train (-1), test (-1)
• STEAD — total 1000000; splits: train (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Computed by plotting the true positive rate against the false positive rate at various threshold levels for the binary event detection task.
• accuracy — range: [0, 1]
  • Fraction of correctly classified phase identification samples. Determined by comparing the peak P-phase and S-phase probabilities to assign a class label.

Input / output format

Input: 30-second seismic waveform windows randomly sampled from continuous traces.

Output: 3-dimensional probability vector per timestep representing [noise, P-phase, S-phase] probabilities, or class probabilities for classification heads.

Scoring recipe

def score_event_detection(probs, y_true):
    event_score = 1.0 - probs[:, 0]  # 1 - noise probability
    return roc_auc_score(y_true, event_score)

def score_phase_identification(probs, y_true):
    p_score, s_score = probs[:, 1], probs[:, 2]
    pred = np.where(p_score > s_score, 'P', 'S')
    return accuracy_score(y_true, pred)

def score_onset_regression(probs, y_true):
    axis=1)
    return mean_absolute_error(y_true, onset_time)

Common pitfalls

• Using 1 - noise_probability for event detection instead of a dedicated binary classification head.
• Evaluating across varying training data fractions (5% to 100%) rather than a fixed held-out test set.
• Phase identification requires comparing P and S peak probabilities, not taking argmax over the full 3-class vector.

Evidence (verbatim from paper)

For a quantitative analysis, we consider the three evaluation tasks defined in the large-scale benchmark by Münchmeyer et al. (2022): 1. Event detection: Given a window of a seismic waveform, determine if it contains an event. 2. Phase identification: Given a window containing exactly one phase arrival, determine if it is a P or an S phase. ... The y axis shows the AUC metric: it represents the area under the curve that plots the true positive rate against the false positive rate at various threshold levels for a binary classification task.

Citation

@misc{liu2024seislm,
  title={SeisLM: a Foundation Model for Seismic Waveforms},
  author={Tianlin Liu et al. (2024)},
  year={2024},
  note={arXiv:2410.15765}
}

• arXiv: 2410.15765