Tabular Attention Vs Contrastive Eval

tabular-attention-vs-contrastive-eval

Attention versus Contrastive Learning of Tabular Data -- A Data-centric Benchmarking — Rabbani et al. (2024) (arXiv:2401.04266, 2024)

What this evaluates

Evaluates the performance of traditional machine learning, deep learning, attention-based, and contrastive learning methods on tabular classification tasks. It probes how data characteristics (dimensionality, difficulty) influence the optimal learning strategy and compares different masking/filling strategies used in contrastive learning.

Datasets

• OpenML Tabular Benchmark — total ?; splits: test (-1)

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall for classification. Reported as mean scores with standard deviations in parentheses across multiple runs.
• Win ratio — range: [0, 28]
  • Count of datasets where method A statistically outperforms method B, determined via Wilcoxon-signed rank test.

Input / output format

Input: Tabular feature matrix containing numerical and categorical columns with corresponding class labels.

Output: Predicted class labels or class probabilities for each instance.

Scoring recipe

# For each of the 28 datasets:
# 1. Train each model (LR, GBT, DNN, DNN-AE, TabNet, FTT, NPT, Pass, Noise, Sample, CutMix, RFC, WCR, SAINT)
# 2. Generate predictions on the test set
# 3. Compute F1 score for each model
# 4. Rank models by F1 score (1 = best)
# 5. Average ranks across all 28 datasets
# 6. Resolve ties using lower standard deviation of F1 scores
# 7. Apply Wilcoxon-signed rank test for pairwise statistical significance

Common pitfalls

• Several deep models (NPT, TabNet) run out of memory on specific datasets, requiring exclusion from rank averages for those cases.
• Rank ties are broken by comparing the standard deviation of F1 scores across runs, not just mean scores.
• Statistical significance is assessed via Wilcoxon-signed rank test, not just raw F1 differences.

Evidence (verbatim from paper)

CutMix yields the highest F1 scores on eight out of 28 data sets. Among these eight data sets, hard and easy-to-classify datasets are evenly split. RFC also shows the best F1 scores on eight other data sets, where hard and easy datasets are in a six-to-two ratio. The average rank (AvgRank) in Table 6 across 28 data sets reveals that CutMix (AvgRank: 6.75) and RFC (AvgRank: 6.96) are superior to traditional data corruption methods, including adding noise (Noise, AvgRank: 8.89) and sampling from distribution (Sample, AvgRank: 7.11).

Citation

@misc{rabbani2024attention,
  title={Attention versus Contrastive Learning of Tabular Data -- A Data-centric Benchmarking},
  author={Rabbani et al. (2024)},
  year={2024},
  note={arXiv:2401.04266}
}

• arXiv: 2401.04266