Tabzilla Hard Reval

tabzilla-hard-reval

Unreflected Use of Tabular Data Repositories Can Undermine Research Quality — Tschalzev et al. (2025) (arXiv:2503.09159, 2025)

What this evaluates

This protocol re-evaluates tabular benchmarks to measure how validation strategy (holdout vs. 5-fold cross-validation) and hyperparameter optimization budgets affect model selection and reported performance. It probes the robustness of empirical conclusions in tabular machine learning when standard holdout validation is replaced with cross-validation ensembles.

Datasets

• TabZilla-hard — total 36; splits: train (-1), val (-1), test (-1)
• Grinsztajn et al. (2022) benchmark — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• logloss (primary) — range: other
  • Log-loss (cross-entropy loss) computed on the test set. For binary classification: -1/N * Σ(y_i log(p_i) + (1-y_i) log(1-p_i)). For multi-class, it is the categorical cross-entropy averaged over samples.

Input / output format

Input: Tabular feature matrix and target vector, partitioned into train, validation, and test splits as provided by the benchmark.

Output: Predicted probabilities for each class per test instance, used to compute test logloss.

Scoring recipe

def compute_logloss(y_true, y_pred_proba):
    y_pred = np.clip(y_pred_proba, 1e-15, 1 - 1e-15)
    if y_true.ndim == 1:
        return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))
    else:
        return -np.mean(np.log(y_pred[np.arange(len(y_true)), y_true]))

# For 5CV protocol:
predictions = []
for fold in range(5):
    model = train(train_fold, val_fold)
    predictions.append(model.predict_proba(test_fold))
avg_pred = np.mean(predictions, axis=0)
score = compute_logloss(y_test, avg_pred)

Common pitfalls

• Using single holdout validation for hyperparameter tuning often overfits to the validation set, leading to biased model selection and underestimated test performance.
• Comparing new methods against precomputed benchmark results without re-tuning strong baselines (like XGBoost/MLP) yields false SOTA claims.
• Assuming holdout validation is universally appropriate ignores dataset size effects; small datasets benefit significantly from 5-fold cross-validation ensembles.

Evidence (verbatim from paper)

For each fold/split, the hyperparameters of each model are independently tuned over 100 trials. In line with related work, the best model is selected on the validation data. For the TabZilla datasets, we additionally evaluate an alternative model selection procedure where, instead of the default holdout validation split, 5-fold cross-validation (5CV) is used. In this regime, we obtain predictions by averaging the predictions of all fold models... Figure 2 shows that with the holdout validation strategy, it is more likely to miss the hyperparameters with stronger performance... cumulative density functions of test logloss performance over 100 trials for MLPs and XGBoost.

Citation

@misc{tschalzev2025unreflected,
  title={Unreflected Use of Tabular Data Repositories Can Undermine Research Quality},
  author={Tschalzev et al. (2025)},
  year={2025},
  note={arXiv:2503.09159}
}

• arXiv: 2503.09159