Nasadat Covid Severity Eval

nasadat-covid-severity-eval

Using NASA Satellite Data Sources and Geometric Deep Learning to Uncover Hidden Patterns in COVID-19 Clinical Severity — Ignacio Segovia-Dominguez et al. (2021) (arXiv:2110.10849, 2021)

What this evaluates

Evaluates the ability of deep learning and geometric deep learning models to forecast county-level COVID-19 hospitalizations using satellite-derived atmospheric variables (AOD, temperature, humidity) alongside baseline features. It probes spatio-temporal forecasting capabilities and the conditional predictive utility of environmental risk factors on disease severity.

Datasets

• NASAdat — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• RMSE (primary) — range: [0, inf)
  • Root Mean Squared Error between predicted and actual COVID-19 hospitalization counts. Calculated as the square root of the average of squared differences between forecasts and true records over the evaluation period.

Input / output format

Input: County-level time series of atmospheric variables (Aerosol Optical Depth, surface temperature, relative humidity) and baseline features, structured for spatio-temporal graph or sequence modeling.

Output: Predicted number of COVID-19 hospitalizations for a 15-day ahead forecasting horizon per county.

Scoring recipe

def compute_rmse(predictions, gold):
    import numpy as np
    predictions = np.array(predictions)
    gold = np.array(gold)
    squared_diff = (predictions - gold) ** 2
    mean_squared_error = np.mean(squared_diff)
    return np.sqrt(mean_squared_error)

Common pitfalls

• The evaluation uses a fixed 15-day ahead forecasting horizon, not multi-step or point-in-time predictions.
• RMSE is explicitly chosen over R^2 for primary validation, despite R^2 being common in epidemiology; results should not be compared directly to R^2 benchmarks.
• Models are evaluated on the conditional predictive utility of atmospheric variables, meaning socio-demographic and mobility factors are held fixed or excluded.

Evidence (verbatim from paper)

In our experiments we use the RMSE metric rather than $R^{2}$ since RMSE is the standard metric for validation of predictive models in space-time forecasting. Despite statistical criticism, $R^{2}$ is still used in epidemiology. As such, we present a summary of results for $R^{2}$. While we find that $R^{2}$ for actual observations and hospitalization forecasts with/without AOD are generally similar in CA, in TX and PA $R^{2}$ for GCNs with AOD tends to be from 0.05 to 0.25 higher than $R^{2}$ for the same GCN but without AOD, with ranges from 0.6 to 0.88 in PA and from 0.71 to 0.93 in TX. These findings echo our conclusions on contributions of AOD to COVID-19 clinical severity, based on predictive RMSE.

Citation

@misc{segovia2021nasadat,
  title={Using NASA Satellite Data Sources and Geometric Deep Learning to Uncover Hidden Patterns in COVID-19 Clinical Severity},
  author={Ignacio Segovia-Dominguez et al. (2021)},
  year={2021},
  note={arXiv:2110.10849}
}

• arXiv: 2110.10849