Patient Flow Forecasting Eval

patient-flow-forecasting-eval

Forecasting Patient Flows with Pandemic Induced Concept Drift using Explainable Machine Learning — Susnjak et al. (2022) (arXiv:2211.00739, 2022)

What this evaluates

Evaluates the ability of machine learning and statistical models to forecast daily patient flows at urgent care clinics, specifically testing robustness to concept drift induced by pandemic disruptions using quasi-real-time proxy variables.

Datasets

• Clinic 1 & 2 patient flow data — total ?; splits: full (-1)

Metrics

• MAPE (primary) — range: percent
  • Mean Absolute Percentage Error. Calculated as the average of the absolute percentage errors between predicted and actual patient counts across the evaluation period. Lower values indicate better accuracy.
• RMSE — range: other
  • Root Mean Squared Error. The square root of the average of squared differences between predicted and actual values.
• MAE — range: other
  • Mean Absolute Error. The average of absolute differences between predicted and actual values.

Input / output format

Input: Daily historical patient counts (autoregressive lags) combined with quasi-real-time proxy features (Google search trends, pedestrian traffic, flu incidence, COVID-19 Alert Levels).

Output: Forecasted daily patient flow count for the target clinic.

Scoring recipe

def calculate_mape(gold, predictions):
    mask = gold != 0
    return np.mean(np.abs((gold[mask] - predictions[mask]) / gold[mask])) * 100

def calculate_rmse(gold, predictions):
    return np.sqrt(np.mean((gold - predictions) ** 2))

def calculate_mae(gold, predictions):
    return np.mean(np.abs(gold - predictions))

Common pitfalls

• MAPE is highly sensitive to low patient volumes, making direct accuracy comparisons across clinics with different baseline volumes misleading.
• Evaluating performance solely on overall averages masks the severe, non-stationary drops in accuracy caused by pandemic-induced concept drift; year-by-year analysis is required.
• The in-house benchmark model is not a standard statistical baseline, so reported percentage improvements over it may not generalize to other forecasting approaches.

Evidence (verbatim from paper)

Table 4 shows a high-level summary of all the models across both clinics, displaying the MAPE values for each candidate model developed with a full set of features. ... This result underscores the limitations of blindly using the MAPE measure for comparisons across different studies without taking volumes into consideration since the magnitude of the total patient volumes affects variability and consequently, predictability.

Citation

@misc{susnjak2022forecasting,
  title={Forecasting Patient Flows with Pandemic Induced Concept Drift using Explainable Machine Learning},
  author={Susnjak et al. (2022)},
  year={2022},
  note={arXiv:2211.00739}
}

• arXiv: 2211.00739