Grinsztajn45 Eval

grinsztajn45-eval

Trompt: Towards a Better Deep Neural Network for Tabular Data — Chen et al. (2023) (arXiv:2305.18446, 2023)

What this evaluates

Evaluates the predictive performance of deep neural networks on tabular data across classification and regression tasks, comparing them against tree-based models and other DNNs. It probes how well architectures handle numerical-only versus heterogeneous (numerical + categorical) features at different dataset scales.

Datasets

• Grinsztajn45 — total ?; splits: train (-1), test (-1); repo https://github.com/LeoGrin/tabular-benchmark

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: fraction of correctly predicted class labels out of total test samples.
• r2-score — range: [0, 1]
  • Standard coefficient of determination for regression: 1 - (sum of squared residuals / total sum of squares).

Input / output format

Input: Tabular dataset rows containing numerical and/or categorical features, normalized according to the Grinsztajn45 benchmark protocol.

Output: Predicted class labels for classification tasks, or continuous values for regression tasks.

Scoring recipe

if task == 'classification':
    acc = sum(pred == gold) / len(gold)
elif task == 'regression':
    ss_res = sum((gold - pred) ** 2)
    ss_tot = sum((gold - mean(gold)) ** 2)
    r2 = 1 - (ss_res / ss_tot)
return acc if classification else r2

Common pitfalls

• The original Grinsztajn45 benchmark omitted some baseline models due to incomplete results; this paper adds two tree-based models for comparison, which may skew direct head-to-head baselines.
• Trompt uses a smaller hyperparameter search space than competing models, meaning performance gaps may partly reflect search budget rather than pure architectural superiority.
• Results are aggregated and reported by dataset size (medium/large) and feature type (numerical only vs. heterogeneous), so reporting raw per-dataset scores without this grouping will not match the paper's figures.

Evidence (verbatim from paper)

The performance and ablation study of Trompt primarily focus on the Grinsztajn45 benchmark (Grinsztajn et al., 2022). This benchmark comprises datasets from various domains and follows a unified methodology for evaluating different models, providing a fair and comprehensive assessment. The evaluation metrics are accuracy and r2-score for classification and regression tasks, respectively.

Citation

@misc{chen2023trompt,
  title={Trompt: Towards a Better Deep Neural Network for Tabular Data},
  author={Chen et al. (2023)},
  year={2023},
  note={arXiv:2305.18446}
}

• arXiv: 2305.18446