Tabfsbench Eval

tabfsbench-eval

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments — Cheng et al. (2025) (arXiv:2501.18935, 2025)

What this evaluates

Evaluates the robustness of tabular machine learning models when the set of available features dynamically changes in open environments. It measures performance degradation across classification and regression tasks under varying degrees of feature removal (20% to 100%).

Datasets

• TabFSBench datasets — total ?; splits: train&validation (-1), test (-1); repo https://github.com/LAMDASZ-ML/TabFSBench

Metrics

• accuracy — range: [0, 1]
  • Standard classification accuracy (fraction of correct predictions).
• ROC-AUC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve.
• RMSE — range: other
  • Root Mean Square Error for regression tasks; lower values indicate better performance.
• performance gap (Δ) (primary) — range: percent
  • Percentage of model performance gap under feature shifts: Δ = (metric_i - metric_0) / metric_0, where metric_i is the performance with i features shifted and metric_0 is the baseline performance.

Input / output format

Input: Tabular feature matrix with target labels. For LLM-based models, features are formatted into a 'List Template' (structured list-based text input).

Output: Class labels (for classification) or continuous values (for regression).

Scoring recipe

# Compute baseline metric on full feature set
metric_0 = compute_metric(y_true, y_pred_full)

# Compute metric on shifted feature set
metric_i = compute_metric(y_true_shifted, y_pred_shifted)

# Calculate performance gap Δ
delta = (metric_i - metric_0) / metric_0

# For classification: use accuracy or ROC-AUC
# For regression: use RMSE (note: lower RMSE is better, so Δ may be negative for degradation)

Common pitfalls

• The performance gap Δ formula assumes higher metric values indicate better performance; for RMSE (where lower is better), the sign of Δ may need inversion or careful interpretation to correctly reflect degradation.
• Feature shift degrees are fixed at 20%, 40%, 60%, 80%, and 100%; models must be evaluated at each specific degree rather than averaging across arbitrary shifts.
• LLMs require the specific 'List Template' formatting to parse tabular data effectively; standard CSV or JSON inputs may yield poor results.

Evidence (verbatim from paper)

For classification tasks, we utilize accuracy and ROC-AUC as model performance, where higher values denote superior model performance. For regression tasks, we utilize Root Mean Square Error (RMSE) as model performance, where lower values denote superior model performance. We also consider the percentage of model performance gap Δ as model robustness in feature-shift scenarios, $$ \Delta = \frac {\left(m e t r i c _ {i} - m e t r i c _ {0}\right)}{m e t r i c _ {0}} $$ metric $_i$ denotes the model performance where $i$ features shift.

Citation

@misc{cheng2025tabfsbench,
  title={TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments},
  author={Cheng et al. (2025)},
  year={2025},
  note={arXiv:2501.18935}
}

• arXiv: 2501.18935