Lightning Wildfire Prediction Eval

lightning-wildfire-prediction-eval

Global Lightning-Ignited Wildfires Prediction and Climate Change Projections based on Explainable Machine Learning Models — Shmuel et al. (2024) (arXiv:2409.10046, 2024)

What this evaluates

Evaluates machine learning models' ability to classify lightning-ignited wildfire occurrences versus non-occurrences using meteorological, vegetation, and spatio-temporal features. It probes the models' generalization across different feature configurations and geographic regions, highlighting the necessity of separate models for lightning versus anthropogenic fires.

Datasets

• Global Lightning-Ignited Wildfire Dataset — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • The proportion of correctly classified instances (ignition vs. non-ignition) out of the total test set instances.
• ROC AUC — range: [0, 1]
  • The area under the Receiver Operating Characteristic curve, measuring the model's ability to distinguish between classes across all classification thresholds.
• F1 Score — range: [0, 1]
  • The harmonic mean of precision and recall, providing a single metric that balances both false positives and false negatives.
• Precision — range: [0, 1]
  • The ratio of true positive predictions to the total number of positive predictions made by the model.
• Recall — range: [0, 1]
  • The ratio of true positive predictions to the total number of actual positive instances in the dataset.

Input / output format

Input: Tabular/spatio-temporal feature vectors including vegetation cover, meteorological variables (e.g., RH, temperature, wind velocity), fire weather indices (FWI, FFMC), historical burned area, and geographic coordinates.

Output: Binary classification label (ignition vs. non-ignition) or predicted probability of ignition.

Scoring recipe

def compute_metrics(y_true, y_pred, y_prob=None):
    accuracy = (y_true == y_pred).mean()
    metrics = {'accuracy': accuracy}
    if y_prob is not None:
        metrics['auc'] = roc_auc_score(y_true, y_prob)
    metrics['precision'] = precision_score(y_true, y_pred)
    metrics['recall'] = recall_score(y_true, y_pred)
    metrics['f1'] = f1_score(y_true, y_pred)
    return metrics

Common pitfalls

• Applying a model trained on anthropogenic wildfires to predict lightning-ignited fires (or vice versa) yields significantly lower accuracy (~70-75% vs ~89-91%).
• Assuming linear feature-target relationships; the data exhibits strong non-linear dependencies, causing linear models like Logistic Regression to underperform tree-based ensembles.
• Omitting spatio-temporal data reduces model accuracy by ~2.3-3.7% across all algorithms.

Evidence (verbatim from paper)

Table 2 shows the performance of four ML models - Logistic Regression, Random Forest, AutoGluon, and XGBoost for five different configurations of source features, divided into four sets - vegetation, meteorological, and anthropogenic factors, fire history, FWIs, and spatio-temporal data. The results for each feature configuration and model are shown as the accuracy of the obtained model on the test set. For all five feature configurations, a clear order of performance emerges where the XGboost model provides the highest accuracy followed by the AutoGluon, Random Forest, and Logistic Regression models. In particular, with all the features in the dataset, the XGboost obtains a 91.6% accuracy, compared to the benchmark of a logistic regression with an accuracy of 82.1%, a 9.5% performance increase.

Citation

@misc{shmuel2024lightningwildfire,
  title={Global Lightning-Ignited Wildfires Prediction and Climate Change Projections based on Explainable Machine Learning Models},
  author={Shmuel et al. (2024)},
  year={2024},
  note={arXiv:2409.10046}
}

• arXiv: 2409.10046