Wild Tab Eval

wild-tab-eval

Wild-Tab: A Benchmark For Out-Of-Distribution Generalization In Tabular Regression — Kolesnikov (2023) (arXiv:2312.01792, 2023)

What this evaluates

Evaluates the out-of-distribution (OOD) generalization capability of tabular regression models by measuring performance gaps between in-distribution and out-of-distribution test sets. It probes whether advanced OOD training strategies or complex architectures can reliably outperform simple Empirical Risk Minimization (ERM) on unseen data distributions.

Datasets

• VPower_S — total ?; splits: train (-1), val (-1), test (-1)
• VPower_R — total ?; splits: train (-1), val (-1), test (-1)
• Weather — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• MAE (primary) — range: other (kW or Celsius)
  • Mean Absolute Error: the average of the absolute differences between predicted and true values. Formula: MAE = (1/N) * Σ|y_true - y_pred|.
• RMSE — range: other (kW or Celsius)
  • Root Mean Squared Error: the square root of the average of squared differences between predicted and true values.
• MAPE — range: percent
  • Mean Absolute Percentage Error: the average of absolute percentage errors between predicted and true values.

Input / output format

Input: Tabular feature vectors containing numerical and vector-based features representing real-world industrial data (e.g., weather conditions, power consumption metrics).

Output: Continuous scalar regression target (e.g., power consumption in kW or temperature in Celsius).

Scoring recipe

def compute_mae(y_true, y_pred):
    import numpy as np
    return np.mean(np.abs(np.array(y_true) - np.array(y_pred)))

Common pitfalls

• Validation performance on OOD data does not reliably forecast OOD test performance; strong validation results do not ensure robust OOD generalization.
• Advanced OOD methods (e.g., IRM, IB-ERM) are highly sensitive to hyperparameter tuning and often fail to outperform simple Empirical Risk Minimization (ERM) without careful configuration.
• Using average-in-domain validation for model selection can mislead performance rankings compared to average-out-domain validation, which is better suited for OOD tasks.

Evidence (verbatim from paper)

Each dataset’s performance is measured via the MAE, expressed in kW for $VPower_{S}$ and $VPower_{R}$ datasets and in Celsius for the $Weather$ dataset. Supplementary results using RMSE and MAPE metrics are available in Appendix C. In all scenarios, smaller figures denote superior outcomes.

Citation

@misc{kolesnikov2023wildtab,
  title={Wild-Tab: A Benchmark For Out-Of-Distribution Generalization In Tabular Regression},
  author={Kolesnikov (2023)},
  year={2023},
  note={arXiv:2312.01792}
}

• arXiv: 2312.01792