Clear Regression Eval

clear-regression-eval

CLeaR: An Adaptive Continual Learning Framework for Regression Tasks — He et al. (2021) (arXiv:2101.00926, 2021)

What this evaluates

Evaluates a model's ability to continuously learn from non-stationary data streams without catastrophic forgetting, balancing stability and plasticity in regression tasks.

Datasets

• Artificial periodic dataset — total 12000; splits: warm-up (1000), update (10000), evaluation (1000)
• Wind power generation dataset — total ?; splits: warm-up (1000), update (10000), evaluation (1000)

Metrics

• fitting error — range: other
  • Mean Squared Error (MSE) computed over the 11,000 samples from the warm-up and update phases. Lower values indicate better knowledge accumulation.
• prediction error (primary) — range: other
  • Mean Squared Error (MSE) computed over the 1,000 samples in the evaluation phase. Reflects generalization to unseen data.
• forgetting ratio — range: other
  • (max(0, L_warm_up^2 - L_warm_up^1)) / L_warm_up^1, where L^1 is MSE on warm-up data after the warm-up phase, and L^2 is MSE on the same data after the update phase.

Input / output format

Input: 7-dimensional feature vectors (e.g., meteorological forecasts or synthetic periodic signals).

Output: Continuous scalar regression target (e.g., normalized wind power generation or reconstructed input).

Scoring recipe

# Fitting Error (on 11,000 seen samples)
fitting_error = np.mean((pred_seen - target_seen) ** 2)

# Prediction Error (on 1,000 evaluation samples)
pred_error = np.mean((pred_eval - target_eval) ** 2)

# Forgetting Ratio (only for updated models)
forgetting_ratio = max(0, mse_warmup_after_update - mse_warmup_after_warmup) / mse_warmup_after_warmup

Common pitfalls

• Fitting error is calculated on seen data (warm-up + update), while prediction error is strictly on the held-out evaluation phase.
• Forgetting ratio is only defined for models that undergo updates (Instance B and C); it is not applicable to the baseline or Instance A.
• The artificial dataset uses an unsupervised reconstruction objective, whereas the wind power dataset uses a supervised regression objective, requiring different model heads.

Evidence (verbatim from paper)

Models are evaluated in terms of fitting error, prediction error, and forgetting ratio. Fitting error indicates how well the instance fits all seen samples after the update phase. Updating a CLeaR instance by mini-batch data might lead to a local minimum during the updating process. Eventually, the instance fits only a specific subset rather than all seen data. Such effects are measured by calculating the MSE on the 11000 samples, 1000 of which are from the warm-up phase, and the rest 10000 are from the update phase. Therefore, a lower fitting error reflects that more knowledge is finally accumulated. Prediction error reflects the ability of CLeaR instances to perform predictions on previously unseen data. It is calculated with 1000 samples in the evaluation phase. Forgetting ratio measures how much old knowledge a model forgets after learning new tasks. The formula is forgetting ratio = max(0, L_warm_up^2 - L_warm_up^1) / L_warm_up^1, where L_warm_up^1 indicates the MSE on the warm-up dataset at the end of the warm-up phase and L_warm_up^2 indicates the error on the same dataset at the end of the update phase, and max(x1, x2) returns the larger one of either x1 or x2.

Citation

@misc{he2021clear,
  title={CLeaR: An Adaptive Continual Learning Framework for Regression Tasks},
  author={He et al. (2021)},
  year={2021},
  note={arXiv:2101.00926}
}

• arXiv: 2101.00926