Earthquake Detection Eval

earthquake-detection-eval

Deep vs. Shallow Learning: A Benchmark Study in Low Magnitude Earthquake Detection — Goel et al. (2022) (arXiv:2205.00525, 2022)

What this evaluates

Binary classification of low-magnitude seismic events versus background noise in seismological time-series data. It probes model robustness to varying noise-to-signal ratios and evaluates the trade-off between detection sensitivity and false positive rates in safety-critical monitoring.

Datasets

• Groningen gas field seismic data — total ?; splits: test (-1)

Metrics

• MCC (primary) — range: [-1, 1]
  • Matthews Correlation Coefficient, calculated as (TP×TN−FP×FN)/√((TP+FP)(TP+FN)(TN+FP)(TN+FN)). Ranges from -1 to 1, with 1 indicating perfect prediction.
• Accuracy — range: [0, 1]
  • Proportion of correct predictions: (TP+TN)/(TP+TN+FP+FN).

Input / output format

Input: Seismological time-series recordings, either processed into statistical features (e.g., 26 features from HCTSA and catch22 for the LR model) or raw/pre-trained representations for the CNN.

Output: Binary class label: 'earthquake' (positive) or 'noise' (negative).

Scoring recipe

def compute_metrics(y_true, y_pred):
    tp = sum((t == 1) and (p == 1) for t, p in zip(y_true, y_pred))
    tn = sum((t == 0) and (p == 0) for t, p in zip(y_true, y_pred))
    fp = sum((t == 0) and (p == 1) for t, p in zip(y_true, y_pred))
    fn = sum((t == 1) and (p == 0) for t, p in zip(y_true, y_pred))
    acc = (tp + tn) / (tp + tn + fp + fn)
    denom = ((tp+fp)*(tp+fn)*(tn+fp)*(tn+fn))**0.5
    mcc = (tp*tn - fp*fn) / denom if denom > 0 else 0.0
    return acc, mcc

Common pitfalls

• High accuracy at elevated noise-to-signal ratios can be misleading for the CNN, as it achieves this by perfectly classifying noise (high TN) while missing most earthquakes (high FN).
• Statistical significance testing (at 5% level) is required to claim performance improvements between models, rather than relying solely on point estimates.
• MCC is preferred over accuracy for imbalanced datasets, as accuracy alone does not reflect the trade-off between false positives and false negatives in seismic detection.

Evidence (verbatim from paper)

From Table 1 it can be seen the original LR model is able to achieve an MCC of 0.9691, but that our augmented model is even so able to statistically significantly improve on it. However, the CNN does substantially less well.

Citation

@misc{goel2022deep,
  title={Deep vs. Shallow Learning: A Benchmark Study in Low Magnitude Earthquake Detection},
  author={Goel et al. (2022)},
  year={2022},
  note={arXiv:2205.00525}
}

• arXiv: 2205.00525