Dsp Toxicity Prediction Eval

dsp-toxicity-prediction-eval

Explainable machine learning for predicting shellfish toxicity in the Adriatic Sea using long-term monitoring data of HABs — Marzidovšek et al. (2024) (arXiv:2405.04372, 2024)

What this evaluates

Evaluates machine learning models' ability to predict diarrhetic shellfish poisoning (DSP) toxicity events in mussels using long-term environmental and phytoplankton monitoring data. It probes the model's capacity to integrate biological indicators (toxic species abundance) with abiotic drivers (salinity, river flow, temperature) for binary hazard forecasting.

Datasets

• Gulf of Trieste HAB monitoring dataset — total ?; splits: train (-1), test (-1); repo https://github.com/MartinMarzi/HABTox-predictor

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: F1 = 2 * (precision * recall) / (precision + recall). Optimized during hyperparameter tuning via grid search over 5 folds.
• Precision — range: [0, 1]
  • Ratio of correctly predicted positive toxicity instances to all instances predicted as positive: TP / (TP + FP).
• Recall — range: [0, 1]
  • Ratio of correctly predicted positive toxicity instances to all actual positive instances: TP / (TP + FN).

Input / output format

Input: Tabular features including abundances of DSP-producing phytoplankton species (e.g., Dinophysis fortii, D. caudata), 90th percentile DSP toxin concentrations (DSP-tot), and environmental variables (surface salinity, river discharge, precipitation, air temperature, solar radiation duration).

Output: Binary classification label: 'poz' (positive toxicity result) or 'neg' (negative toxicity result).

Scoring recipe

def compute_metrics(y_true, y_pred):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 1)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return {'precision': prec, 'recall': rec, 'f1': f1}

final_metrics = {'precision': 0, 'recall': 0, 'f1': 0}
for _ in range(100):
    X_train, X_test, y_train, y_test = split_data()
    X_train_bal = apply_smote_and_under_sampler(X_train, y_train)
    model = train_and_grid_search(X_train_bal, y_train, metric='f1')
    y_pred = model.predict(X_test)
    scores = compute_metrics(y_test, y_pred)
    for k in final_metrics: final_metrics[k] += scores[k]
for k in final_metrics: final_metrics[k] /= 100

Common pitfalls

• The dataset is highly imbalanced; the authors apply both SMOTE and RandomUnderSampler, which can artificially inflate performance if not carefully validated across the 100 iterations.
• Models are explicitly optimized for F1 score, which causes a steep drop in precision at higher recall values (>0.6), making the precision-recall trade-off critical for real-world early warning systems.
• Small dataset size leads to spurious negative permutation importance values for irrelevant features, which may be misinterpreted as meaningful negative correlations rather than chance artifacts.

Evidence (verbatim from paper)

Using the ML pipeline, SVM, DT, RF and ANN models were trained to predict DSP toxicity, with hyperparameters individually optimised using grid search based on the mean F1 score over 5 folds. To obtain a reliable estimate of how the models would perform with similar data in the real world, the entire pipeline (model construction with hyperparameter optimization on the training set and evaluation on the test set) was run through 100 iterations and the results averaged. This was important due to the variability of results between runs and between folds (Figure 6) and allowed for a more reliable algorithm comparison using the three performance metrics of precision, recall, and F1 score.

Citation

@misc{marzidovsek2024habtox,
  title={Explainable machine learning for predicting shellfish toxicity in the Adriatic Sea using long-term monitoring data of HABs},
  author={Marzidovšek et al. (2024)},
  year={2024},
  note={arXiv:2405.04372}
}

• arXiv: 2405.04372