Lm Weather Eval

lm-weather-eval

Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Models — Chen et al. (2024) (arXiv:2405.20348, 2024)

What this evaluates

Evaluates on-device meteorological variable forecasting and imputation capabilities using a federated learning framework with personalized adapters. It tests the model's ability to predict regional weather trends and handle missing data under data scarcity and heterogeneous distributions.

Datasets

• On-device Weather Series (ODW1/ODW2) — total ?; splits: test (-1)

Metrics

• MAE (primary) — range: other (lower is better)
  • Mean Absolute Error: average of absolute differences between predicted and actual values. Lower is better.
• RMSE — range: other (lower is better)
  • Root Mean Square Error: square root of the average of squared differences between predicted and actual values. Lower is better.

Input / output format

Input: Multivariate meteorological time series sequences of length 192 collected from regional weather stations.

Output: Forecasted values for prediction horizons of 96, 192, 336, or 720 time steps, or imputed values for randomly masked time steps.

Scoring recipe

def compute_metrics(preds, targets):
    mae = np.mean(np.abs(preds - targets))
    rmse = np.sqrt(np.mean((preds - targets) ** 2))
    return mae, rmse

Common pitfalls

• Lower MAE/RMSE values indicate better performance, contrary to accuracy-based metrics.
• All reported results are averaged over 5 independent experimental runs.
• Baseline models are evaluated in a Federated Learning setting using FedAvg aggregation, not standard centralized training.
• Imputation results in the main text specifically use a 50% random masking ratio.

Evidence (verbatim from paper)

Evaluation metrics include mean absolute error (MAE) and root square mean error (RMSE). All our experiments are repeat five times and we report the averaged results. In imputation, we use sequence lengths of {96, 192, 336, 720} and apply three different masking probabilities {25%, 35%, 50%} to represent missing data. The main manuscript shows imputation results for a 50% masking ratio.

Citation

@misc{chen2024personalized,
  title={Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Models},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2405.20348}
}

• arXiv: 2405.20348