Ch4 Detection Intensity Eval

ch4-detection-intensity-eval

Development and Evaluation of Ensemble Learning-based Environmental Methane Detection and Intensity Prediction Models — Majumder et al. (2023) (arXiv:2312.10879, 2023)

What this evaluates

Evaluates ensemble machine learning models for binary detection of fugitive methane emissions and continuous prediction of their tracer concentration intensity using meteorological data.

Datasets

• HYSPLIT-generated Methane Emission Dataset — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Calculated as (TP + TN) / (TP + FP + TN + FN), representing the proportion of correctly classified instances.
• F1-Score — range: [0, 1]
  • Calculated as TP / (TP + (FP + FN) / 2), balancing precision and recall for the positive class.
• MCC — range: [-1, 1]
  • Mathew's Correlation Coefficient: ((TP * TN) - (FP * FN)) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)).
• AUC ROC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, integrating True Positive Rate over False Positive Rate.
• R² — range: (-inf, 1]
  • Coefficient of determination: 1 - (SSR / SST), where SSR is sum of squared residuals and SST is total sum of squares.
• RMSE — range: [0, inf)
  • Root Mean Squared Error: sqrt(1/N * sum((y_true - y_pred)^2)).

Input / output format

Input: 10 meteorological parameters (e.g., wind speed, temperature, humidity, pressure, water vapor, heat flux). Features are standardized for regression models.

Output: Binary 'Leakage' label (0/1) for detection task; continuous tracer concentration value for intensity prediction task.

Scoring recipe

def score_classification(y_true, y_pred):
    tp = sum((y_pred == 1) & (y_true == 1))
    tn = sum((y_pred == 0) & (y_true == 0))
    fp = sum((y_pred == 1) & (y_true == 0))
    fn = sum((y_pred == 0) & (y_true == 1))
    acc = (tp + tn) / (tp + tn + fp + fn)
    f1 = tp / (tp + (fp + fn) / 2)
    return acc, f1

def score_regression(y_true, y_pred):
    ssr = sum((y_true - y_pred) ** 2)
    sst = sum((y_true - mean(y_true)) ** 2)
    r2 = 1 - (ssr / sst)
    rmse = sqrt(mean((y_true - y_pred) ** 2))
    return r2, rmse

Common pitfalls

• Ground truth labels are synthetically generated using the HYSPLIT dispersion model rather than real-world sensor measurements, which may limit field generalizability.
• The paper does not report the exact number of samples or the train/val/test split ratios, making exact replication of dataset size impossible.
• The final ensemble uses weighted averaging of base models, but the specific weights assigned to each base model are not disclosed in the text.

Evidence (verbatim from paper)

For performance evaluation of the classifier models, we used four performance metrics, (i) accuracy (ii) F1-Score, (iii) Mathew's Correlation Coefficient (MCC), and (iv) area under the receiver operating characteristics curve (AUC ROC). ... Our final ensemble model achieved an accuracy of 97.2% on the test dataset and the other performance metrics, such as F1 score, Precision, Recall, MCC, and AUC ROC are 0.972, 0.949, 0.997, 0.945, and 0.995.

Citation

@misc{majumder2023ensemble,
  title={Development and Evaluation of Ensemble Learning-based Environmental Methane Detection and Intensity Prediction Models},
  author={Majumder et al. (2023)},
  year={2023},
  note={arXiv:2312.10879}
}

• arXiv: 2312.10879