Seisclip Eval

seisclip-eval

SeisCLIP: A seismology foundation model pre-trained by multi-modal data for multi-purpose seismic feature extraction — Xu Si et al. (2023) (arXiv:2309.02320, 2023)

What this evaluates

Evaluates a seismology foundation model's ability to classify seismic event types, localize epicenters and depths, and determine focal mechanisms using multi-modal seismic data. It probes cross-dataset generalization and compares fine-tuned, frozen, and scratch-trained variants against spectrum-based baselines.

Datasets

• PNW dataset — total ?; splits: test (-1)
• SCSN dataset — total ?; splits: test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve, computed per class and averaged for macro-average classification performance.
• MAE — range: other
  • Mean Absolute Error between predicted and true values for epicenter distance, depth, coordinates, and magnitude.

Input / output format

Input: Multi-modal seismic data including time-frequency spectra and phase/source information; for focal mechanism analysis, multi-station features are input.

Output: Classification labels (e.g., earthquake, explosion, surface event, or fault type: normal/reverse/strike-slip) or continuous regression values (epicenter distance, depth, coordinates, magnitude).

Scoring recipe

def compute_auc(y_true, y_scores, classes):
    auc_scores = []
    for cls in classes:
        y_bin = (y_true == cls)
        auc_scores.append(roc_auc_score(y_bin, y_scores[cls]))
    return sum(auc_scores) / len(auc_scores)

def compute_mae(y_true, y_pred):
    return mean(abs(y_true - y_pred))

Common pitfalls

• STEAD pre-training dataset only contains earthquakes, creating a domain shift that hurts scratch/frozen models on downstream tasks requiring explosions or surface events.
• Focal mechanism analysis is converted from a complex regression problem (strike, dip, rake) to a simpler 3-class classification due to limited training data, which may obscure fine-grained mechanism estimation.

Evidence (verbatim from paper)

For all classes, the Area Under Curve (AUC) values of the fine-tune model were consistently better than those of the spectrum-based baseline model. Moreover, across all statistical results, the AUC values of the fine-tune model consistently outperformed the frozen model and the model from scratch.

Citation

@misc{si2023seisclip,
  title={SeisCLIP: A seismology foundation model pre-trained by multi-modal data for multi-purpose seismic feature extraction},
  author={Xu Si et al. (2023)},
  year={2023},
  note={arXiv:2309.02320}
}

• arXiv: 2309.02320