Modeconv Anomaly Detection Eval

modeconv-anomaly-detection-eval

ModeConv: A Novel Convolution for Distinguishing Anomalous and Normal Structural Behavior — Schaller et al. (2024) (arXiv:2407.00140, 2024)

What this evaluates

Evaluates a graph neural network's ability to detect structural anomalies by reconstructing multivariate time-series sensor data. It measures how well the model captures physical material properties and eigenmode shifts compared to spectral GNN baselines, while also benchmarking computational efficiency.

Datasets

• Luxembourg dataset — total ?; splits: test (-1)
• Simulated Smart Bridge dataset — total ?; splits: test (-1)

Metrics

• reconstruction error (primary) — range: other
  • The error between the original multivariate time-series sensor input and the autoencoder's reconstructed output. Optimized via hyperparameter search to ensure fair comparison across models.
• training time — range: other
  • Wall-clock duration required to train each model for the fixed number of epochs (50). Used to benchmark computational complexity and runtime efficiency.

Input / output format

Input: Multivariate time-series sensor data structured as a graph, representing physical material properties and sensor correlations over time.

Output: Reconstructed multivariate time-series data and anomaly indicators derived from eigenmode shifts via SVD on modal coordinates.

Scoring recipe

def compute_metrics(predictions, ground_truth, training_time):
    # Calculate reconstruction error (e.g., MSE/MAE between input and output)
    recon_error = np.mean((predictions - ground_truth) ** 2)
    # Record wall-clock training time in seconds
    runtime = training_time
    return {'reconstruction_error': recon_error, 'training_time': runtime}

Common pitfalls

• Hyperparameter tuning was performed via Optuna to minimize reconstruction error across all models, which may obscure dataset-specific optimal configurations.
• Wrappers were manually adapted to make MtGNN and GraphCON compatible with PyTorch Lightning and stable dimension constraints, potentially introducing implementation artifacts.
• Runtime comparisons are hardware-specific (NVIDIA GeForce GTX 1080 Ti) and fixed to 50 epochs, limiting direct cross-platform reproducibility.

Evidence (verbatim from paper)

To validate these claims, we compare the results of a graph autoencoder with ModeConv layers trained on the Luxembourg dataset with the results of the same graph autoencoder using ChebConv layers [[24]], which are a type of Spectral Graph Convolution. We also evaluate the results on the Simulated Smart Bridge dataset. ... All of the models have been tuned via a hyperparameter study using optuna [[4]], that aims to improve the reconstruction error for all networks in order to create a fair experimental setup. ... Furthermore, we compare the training time of all the models and conduct a comparative study on the single blocks of the ModeConv approach.

Citation

@misc{schaller2024modeconv,
  title={ModeConv: A Novel Convolution for Distinguishing Anomalous and Normal Structural Behavior},
  author={Schaller et al. (2024)},
  year={2024},
  note={arXiv:2407.00140}
}

• arXiv: 2407.00140