Tabular Data Centric Eval

tabular-data-centric-eval

A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Data — Tschalzev et al. (2024) (arXiv:2407.02112, 2024)

What this evaluates

This evaluation probes the robustness and relative performance of tabular machine learning models when subjected to expert-level, dataset-specific preprocessing pipelines rather than standardized baselines. It specifically measures how feature engineering, hyperparameter optimization, and test-time adaptation shift model rankings and close performance gaps across real-world competition datasets.

Datasets

• Kaggle competition datasets (MBGM, BPCCM, HQC, SCTP, PSSDP, AEAC, OGPCC, SCS, IFD, SVPC, electricity) — total ?; splits: train (-1), test (-1); repo https://github.com/atschalz/dc_tabeval

Metrics

• leaderboard rank (primary) — range: rank
  • Ordinal position of a model on the Kaggle competition leaderboard, averaged across preprocessing pipelines. Lower rank indicates better performance.
• task metric — range: [0, 1]
  • Numerical performance score reported on the Kaggle leaderboard (e.g., AUC or accuracy). Values range from 0 to 1 in the provided tables.
• Spearman coefficients — range: [-1, 1]
  • Rank correlation coefficient measuring the stability of model rankings across different preprocessing pipelines.

Input / output format

Input: Tabular dataset containing numerical and categorical features with a target variable. Models receive the training split for fitting and the test split for prediction.

Output: Predictions on the test set (class labels or regression values), which are submitted to the Kaggle leaderboard to compute the task metric and rank.

Scoring recipe

def compute_metrics(predictions, gold, all_models_predictions):
    task_scores = {model: compute_auc(gold, preds) for model, preds in all_models_predictions.items()}
    ranks = {model: rankdata(-score, method='min')[0] for model, score in task_scores.items()}
    spearman_corr = spearmanr(ranks_pipeline_a, ranks_pipeline_b).correlation
    return task_scores, ranks, spearman_corr

Common pitfalls

• Assuming datasets are i.i.d. when they often contain temporal characteristics requiring test-time adaptation.
• Using standardized preprocessing pipelines that bias results against models lacking built-in feature engineering (e.g., CatBoost).
• Interpreting leaderboard rankings without filtering out non-expert submissions or accounting for metric saturation in highly competitive competitions.

Evidence (verbatim from paper)

The model rankings change considerably, as indicated by the relatively low Spearman coefficients between the standardized preprocessing pipeline and the other pipelines.

Citation

@misc{tschalzev2024datacentric,
  title={A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Data},
  author={Tschalzev et al. (2024)},
  year={2024},
  note={arXiv:2407.02112}
}

• arXiv: 2407.02112