Seist Earthquake Monitoring Eval

seist-earthquake-monitoring-eval

SeisT: A foundational deep learning model for earthquake monitoring tasks — Sen Li et al. (2023) (arXiv:2310.01037, 2023)

What this evaluates

Evaluates a deep learning model's capability to perform multiple earthquake monitoring tasks, including seismic phase picking, detection, polarity classification, and magnitude estimation. It specifically probes cross-regional out-of-distribution generalization by training on Chinese seismic network data and testing on geologically distinct Pacific Northwest data.

Datasets

• DiTing — total 277000; splits: train (-1), val (-1), test (-1)
• PNW (ComCat event subset) — total 65384; splits: test (65384)

Metrics

• F1-Score (primary) — range: [0, 1] or percent
  • Harmonic mean of Precision and Recall: F1 = 2 * Pr * Re / (Pr + Re). For phase picking, a prediction counts as a true positive only if the residual error is within δ < 0.1s.
• Precision — range: [0, 1]
  • Ratio of true positives to all positive predictions: Pr = Tp / (Tp + Fp).
• Recall — range: [0, 1]
  • Ratio of true positives to all actual positives: Re = Tp / (Tp + Fn).
• MAE — range: [0, ∞)
  • Mean Absolute Error: MAE = (1/N) * sum(|yi - ŷi|). Measures average offset between model predictions and true labels.
• R² — range: (-∞, 1]
  • Coefficient of determination: R² = 1 - sum((yi - ŷi)²) / sum((yi - ȳ)²). Measures correlation between predicted and true values in regression tasks.

Input / output format

Input: Three-component (East, North, Vertical) seismic waveforms, 150–180 seconds long at 50–100 Hz sampling rate, with amplitudes normalized prior to model input.

Output: Task-dependent: probability sequences for phase picking, 0-1 vectors for detection, one-hot vectors for polarity classification, and continuous scalar values for magnitude, back-azimuth, and epicentral distance estimation.

Scoring recipe

def evaluate(preds, golds, task, tol=0.1):
    if task in ['phase_picking', 'detection', 'polarity']:
        tp = sum(1 for p, g in zip(preds, golds) if abs(p - g) < tol)
        fp = sum(1 for p, g in zip(preds, golds) if abs(p - g) >= tol and p == 1)
        fn = sum(1 for p, g in zip(preds, golds) if abs(p - g) >= tol and g == 1)
        pr = tp / (tp + fp) if (tp + fp) > 0 else 0
        re = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * pr * re / (pr + re) if (pr + re) > 0 else 0
        return f1, pr, re
    else: # regression tasks
        mae = mean(abs(preds - golds))
        r2 = 1 - sum((p - g)**2 for p, g in zip(preds, golds)) / sum((g - mean(golds))**2 for g in golds)
        return mae, r2

Common pitfalls

• Phase picking true positives require a strict error tolerance threshold (δ < 0.1s); ignoring this threshold artificially deflates precision and recall.
• The PNW test set is geologically distinct from the training data (DiTing); evaluating on it measures out-of-distribution generalization, not standard i.i.d. performance.
• Label formats are task-specific (Gaussian for phase picking, 0-1 vectors for detection, one-hot for polarity); using mismatched labels during evaluation breaks metric calculation.

Evidence (verbatim from paper)

Various metrics were employed to evaluate the performance of the models on different tasks. The evaluation metrics chosen include Precision (Pr), Recall (Re), F1-Score (F1), Mean Error (Mean), Standard Deviation (Std.), Mean Absolute Error (MAE), and coefficient of determination (R^2)... In the context of phase-picking tasks, samples with larger residuals are considered false positives. Accordingly, samples with residuals within the error tolerance δ < 0.1s were chosen as true positives in this study. The F1-Score is a balanced metric between precision and recall, providing a comprehensive evaluation of the performance of the model.

Citation

@misc{li2023seist,
  title={SeisT: A foundational deep learning model for earthquake monitoring tasks},
  author={Sen Li et al. (2023)},
  year={2023},
  note={arXiv:2310.01037}
}

• arXiv: 2310.01037