Tabular Benchmark Eval

tabular-benchmark-eval

A Closer Look at Deep Learning Methods on Tabular Datasets — Han-Jia Ye et al. (2024) (arXiv:2407.00956, 2024)

What this evaluates

Evaluates the predictive performance and stability of 32 deep learning and tree-based tabular models across a large collection of diverse tabular datasets. It probes how well different architectures handle classification and regression tasks, and how dataset characteristics influence method rankings.

Datasets

• LAMDA-TALENT Benchmark — total 300; splits: binary_classification (100), multi_class_classification (80), regression (120); repo https://github.com/qile2000/LAMDA-TALENT

Metrics

• accuracy — range: [0, 1]
  • Fraction of correctly predicted class labels out of total instances. Standard classification metric.
• RMSE — range: other
  • Root Mean Squared Error: sqrt(mean((y_true - y_pred)^2)). Standard regression metric.
• average_rank (primary) — range: other
  • For each dataset, methods are ranked by accuracy (classification) or RMSE (regression). The average rank is computed across all datasets. Lower values indicate better overall performance.

Input / output format

Input: Tabular feature matrix (numerical/categorical) and corresponding target vector (class labels or continuous values).

Output: Predicted class labels for classification tasks or continuous values for regression tasks.

Scoring recipe

# Per dataset d:
if d.task == 'classification':
    score[d] = accuracy_score(y_true, y_pred)
else:
    score[d] = mean_squared_error(y_true, y_pred, squared=False)

# Aggregate average rank:
ranks = {m: 0 for m in models}
for d in datasets:
    sorted_models = sort_models_by_score(d, ascending=(d.task == 'regression'))
    for rank_idx, model in enumerate(sorted_models):
        ranks[model] += rank_idx + 1
average_rank = {m: ranks[m] / len(datasets) for m in models}

Common pitfalls

• Comparing methods that only support classification (e.g., TabPFN) or only regression (e.g., DNNR) on the full 300-dataset benchmark without filtering for task compatibility.
• Misinterpreting the direction of the average rank metric; lower ranks indicate better performance, not higher.
• Calculating the Tree-DNN score without first normalizing individual method scores per dataset, which skews the performance gap due to varying dataset difficulties.

Evidence (verbatim from paper)

We compare tabular methods across 300 datasets using classification accuracy for classification tasks and RMSE for regression tasks. The average rank of methods over all datasets is calculated following (Delgado et al., 2014), with results shown in Figure 3.

Citation

@misc{ye2024closer,
  title={A Closer Look at Deep Learning Methods on Tabular Datasets},
  author={Han-Jia Ye et al. (2024)},
  year={2024},
  note={arXiv:2407.00956}
}

• arXiv: 2407.00956