Localized Weather Prediction Eval

localized-weather-prediction-eval

Localized Weather Prediction Using Kolmogorov-Arnold Network-Based Models and Deep RNNs — Akazan et al. (2025) (arXiv:2505.22686, 2025)

What this evaluates

Evaluates the ability of neural network architectures (KANs, TKANs, RNNs) to forecast localized weather variables (temperature, precipitation, pressure) one day ahead. Probes nonlinear time-series modeling and regression accuracy under varying data distributions, such as low precipitation versus high temperature variance.

Datasets

• Abidjan — total ?; splits: unspecified (-1); repo https://github.com/AngeClementAkazan/Localized-Weather-Prediction-Using-KAN-and-DeepRNNs
• Kigali — total ?; splits: unspecified (-1); repo https://github.com/AngeClementAkazan/Localized-Weather-Prediction-Using-KAN-and-DeepRNNs

Metrics

• MSE — range: other
  • Mean Squared Error: average of the squares of the errors between predicted and actual values.
• RMSE — range: other
  • Root Mean Squared Error: square root of MSE, providing error in the same units as the target variable.
• MAE — range: other
  • Mean Absolute Error: average of the absolute differences between predicted and actual values.
• R² (primary) — range: [0, 1]
  • Coefficient of Determination: 1 - (SS_res / SS_tot), representing the proportion of variance in the dependent variable predictable from the independent variables.
• MAPE — range: percent
  • Mean Absolute Percentage Error: average of absolute percentage errors between predicted and actual values. Note: highly sensitive to near-zero ground truth values.

Input / output format

Input: Historical time-series weather data (temperature, precipitation, pressure) for a specific city.

Output: Predicted values for temperature (°C), precipitation (mm), and pressure (kPa) for the next day.

Scoring recipe

import numpy as np
def compute_metrics(y_true, y_pred):
    y_true, y_pred = np.array(y_true), np.array(y_pred)
    mse = np.mean((y_true - y_pred) ** 2)
    rmse = np.sqrt(mse)
    mae = np.mean(np.abs(y_true - y_pred))
    ss_res = np.sum((y_true - y_pred) ** 2)
    ss_tot = np.sum((y_true - np.mean(y_true)) ** 2)
    r2 = 1 - (ss_res / ss_tot)
    mape = np.mean(np.abs((y_true - y_pred) / y_true)) * 100
    return {'MSE': mse, 'RMSE': rmse, 'MAE': mae, 'R²': r2, 'MAPE': mape}

Common pitfalls

• MAPE values are heavily inflated for precipitation due to near-zero ground truth values, making it misleading for low-rainfall conditions.
• Model performance varies drastically by variable: KAN dominates temperature (R² > 0.99) but underperforms on pressure compared to standard RNNs.
• No explicit train/validation/test splits or dataset sizes are reported, making reproducibility of split-dependent metrics difficult.

Evidence (verbatim from paper)

The notably high MAPE values across all models are a result of the inherently low precipitation levels in both cities, which inflate relative error calculations.

Citation

@misc{akazan2025localized,
  title={Localized Weather Prediction Using Kolmogorov-Arnold Network-Based Models and Deep RNNs},
  author={Akazan et al. (2025)},
  year={2025},
  note={arXiv:2505.22686}
}

• arXiv: 2505.22686