Bharatbench Eval

bharatbench-eval

BharatBench: Dataset for data-driven weather forecasting over India — Choudhury et al. (2024) (arXiv:2405.07534, 2024)

What this evaluates

Evaluates data-driven machine learning models for medium-range weather forecasting over India. It probes the ability of models to capture spatial and temporal atmospheric dynamics across diverse Indian microclimates for variables like geopotential height, temperature, and precipitation.

Datasets

• BharatBench — total ?; splits: train (-1), test (-1); repo https://github.com/MASLABnitrkl/BharatBench

Metrics

• RMSE (primary) — range: other
  • Root Mean Square Error: the square root of the mean of the squared differences between predicted and observed values. Lower values indicate better accuracy.
• MAE — range: other
  • Mean Absolute Error: the mean of the absolute differences between predicted and observed values. Lower values indicate better accuracy.
• ACC — range: [-1, 1]
  • Anomaly Correlation Coefficient: the Pearson correlation coefficient between predicted and observed anomalies. Higher values indicate better performance.

Input / output format

Input: Spatial grid data (32x32) of meteorological variables (H500, T850, T2m, TP6h) from IMDAA reanalysis.

Output: Predicted spatial grid values for the same variables at 3-day and 5-day lead times.

Scoring recipe

def compute_metrics(y_true, y_pred):
    rmse = np.sqrt(np.mean((y_true - y_pred) ** 2))
    mae = np.mean(np.abs(y_true - y_pred))
    acc = np.corrcoef(y_true.flatten(), y_pred.flatten())[0, 1]
    return {'RMSE': rmse, 'MAE': mae, 'ACC': acc}

Common pitfalls

• ACC is inversely interpreted compared to error metrics; a lower ACC indicates worse performance, not better.
• Forecast errors (RMSE/MAE) increase rapidly up to 7 days before the rate of increase slows down, so lead time selection heavily impacts scores.
• Strong linear relationships between variables (e.g., H500 and T850) can cause simple linear regression to outperform complex neural networks, contrary to typical deep learning expectations.

Evidence (verbatim from paper)

These baseline models tried to predict a variable (H500, T850, T2m, TP6h) three to five days ahead and were evaluated based on RMSE, MAE, and ACC.

Citation

@misc{choudhury2024bharatbench,
  title={BharatBench: Dataset for data-driven weather forecasting over India},
  author={Choudhury et al. (2024)},
  year={2024},
  note={arXiv:2405.07534}
}

• arXiv: 2405.07534