Rf Climate Param Eval

rf-climate-param-eval

Stable machine-learning parameterization of subgrid processes for climate modeling at a range of resolutions — Yuval et al. (2020) (arXiv:2001.03151, 2020)

What this evaluates

Evaluates whether a random forest model can accurately emulate subgrid atmospheric processes (vertical advection, cloud microphysics, turbulent diffusion, surface fluxes, radiative heating) from high-resolution simulation data. It further tests if the learned parameterization enables stable, long-term coarse-resolution climate simulations that reproduce key statistics like mean and extreme precipitation and ITCZ structure compared to high-resolution ground truth.

Datasets

• SAM aquaplanet simulation (high-resolution output) — total ?; splits: train (-1), test (-1)

Metrics

• R^2 (primary) — range: [0, 1]
  • Coefficient of determination: R^2 = 1 - (SS_res / SS_tot), where SS_res is the sum of squared residuals between predicted and true subgrid tendencies, and SS_tot is the total sum of squares relative to the mean of the true values. Measures the proportion of variance in the target explained by the model.
• RMSE — range: other
  • Root mean square error: sqrt(mean((y_true - y_pred)^2)). Measures the standard deviation of prediction errors in the same physical units as the target variable (e.g., kg kg^-1 s^-1 for tendencies, mm/day for precipitation).

Input / output format

Input: Vertical profiles of resolved temperature (T), total non-precipitating water mixing ratio (q_T), precipitating water mixing ratio (q_p), and distance from equator (|y|) for RF-tend; plus lower-tropospheric profiles of T, q_T, zonal wind (u), meridional wind (v), surface wind speed, and |y| for RF-diff.

Output: RF-tend: 144 outputs (48 model levels × 3 variables: subgrid tendencies for h_L, q_T, q_p). RF-diff: 17 outputs (15 model levels of turbulent diffusivity D, plus subgrid surface fluxes for h_L and q_T).

Scoring recipe

import numpy as np
def compute_metrics(y_true, y_pred):
    y_true = np.asarray(y_true)
    y_pred = np.asarray(y_pred)
    ss_res = np.sum((y_true - y_pred) ** 2)
    ss_tot = np.sum((y_true - np.mean(y_true)) ** 2)
    r2 = 1.0 - (ss_res / ss_tot) if ss_tot != 0 else 0.0
    rmse = np.sqrt(np.mean((y_true - y_pred) ** 2))
    return {'R2': r2, 'RMSE': rmse}

Common pitfalls

• Confusing offline performance (R^2/RMSE on withheld data) with online performance (stability and climate statistics in long-term coarse-resolution simulations), as they vary non-monotonically with grid spacing.
• Comparing R^2 across different grid spacings without accounting for the stochastic component that averages out during coarse-graining, which can make larger grid spacings appear better offline despite worse online stability.
• Not coarse-graining the high-resolution ground truth to the same grid spacing as the coarse simulation before calculating extreme precipitation percentiles, leading to unfair comparisons.

Evidence (verbatim from paper)

The offline performance as measured by the coefficient of determination (R^2) improves substantially as the grid spacing increases (Fig. 3a, compare Fig. 3c and 3e), consistent with the idea of more predictable subgrid tendencies with more averaging over larger grid boxes.

Citation

@misc{yuval2020stable,
  title={Stable machine-learning parameterization of subgrid processes for climate modeling at a range of resolutions},
  author={Yuval et al. (2020)},
  year={2020},
  note={arXiv:2001.03151}
}

• arXiv: 2001.03151