Solar Power Forecasting Eval

solar-power-forecasting-eval

Tree-based Forecasting of Day-ahead Solar Power Generation from Granular Meteorological Features — Berlanger et al. (2023) (arXiv:2312.00090, 2023)

What this evaluates

Evaluates tree-based machine learning models for day-ahead solar power generation forecasting at hourly resolution. It probes the models' ability to capture spatial heterogeneity in meteorological features and their robustness under varying weather conditions.

Datasets

• Belgian Solar Power Generation Dataset — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/nberl/tree-based-dah-solar-forecast

Metrics

• RMSE (primary) — range: other
  • Root Mean Squared Error. Computed as the square root of the mean of squared differences between predicted and actual solar generation values.
• MAE — range: other
  • Mean Absolute Error. Computed as the mean of absolute differences between predicted and actual solar generation values.
• SMAPE — range: [0, 1]
  • Symmetric Mean Absolute Percentage Error. Computed as the mean of 2*|y_true - y_pred| / (|y_true| + |y_pred|). The paper sets the error to zero when the denominator is zero.

Input / output format

Input: Hourly meteorological features (e.g., T2m, RH, TCC) and astronomical variables at granular spatial locations in Belgium, used to predict day-ahead solar power generation.

Output: Predicted solar power generation values at hourly resolution.

Scoring recipe

def compute_metrics(y_true, y_pred):
    rmse = np.sqrt(np.mean((y_true - y_pred) ** 2))
    mae = np.mean(np.abs(y_true - y_pred))
    denom = np.abs(y_true) + np.abs(y_pred)
    smape = np.mean(2 * np.abs(y_true - y_pred) / np.where(denom == 0, 1, denom))
    return rmse, mae, smape

Common pitfalls

• SMAPE can become unstable or misleading when actual solar generation values are near zero, as noted in the paper.
• Model Confidence Sets (MCS) are computed separately for each metric, meaning a model might be in the best set for one metric but not another.
• Training times are hardware-dependent and should not be compared across different setups without normalization.

Evidence (verbatim from paper)

Table 3 reports the out-of-sample RMSE, MAE, and SMAPE for the 24 tuned model configurations. A first glance at Table 3 reveals that, regardless of the chosen spatial grid and feature set, the ensemble methods RF and XGBoost obtain better forecast accuracy than the other methods, and this across all forecast metrics.

Citation

@misc{berlanger2023tree,
  title={Tree-based Forecasting of Day-ahead Solar Power Generation from Granular Meteorological Features},
  author={Berlanger et al. (2023)},
  year={2023},
  note={arXiv:2312.00090}
}

• arXiv: 2312.00090