# Tabular Attention Vs Contrastive Eval

> Evaluates the performance of traditional machine learning, deep learning, attention-based, and contrastive learning methods on tabular classification tasks. It probes how data characteristics (dimensionality, difficulty) influence the optimal learning strategy and compares different masking/filling strategies used in contrastive learning. Use when the user wants to benchmark on OpenML Tabular Benchmark, or asks about evaluating this task. Reports F1 score.

- Skill: `qhjqhj00/tabular-attention-vs-contrastive-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/tabular-attention-vs-contrastive-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/tabular-attention-vs-contrastive-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/tabular-attention-vs-contrastive-eval

---


# tabular-attention-vs-contrastive-eval

> Attention versus Contrastive Learning of Tabular Data -- A Data-centric Benchmarking — Rabbani et al. (2024) (arXiv:2401.04266, 2024)

## What this evaluates

Evaluates the performance of traditional machine learning, deep learning, attention-based, and contrastive learning methods on tabular classification tasks. It probes how data characteristics (dimensionality, difficulty) influence the optimal learning strategy and compares different masking/filling strategies used in contrastive learning.

## Datasets

- **OpenML Tabular Benchmark** — total ?; splits: test (-1)

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall for classification. Reported as mean scores with standard deviations in parentheses across multiple runs.
- `Win ratio` — range: [0, 28]
  - Count of datasets where method A statistically outperforms method B, determined via Wilcoxon-signed rank test.

## Input / output format

**Input**: Tabular feature matrix containing numerical and categorical columns with corresponding class labels.

**Output**: Predicted class labels or class probabilities for each instance.

## Scoring recipe

```python
# For each of the 28 datasets:
# 1. Train each model (LR, GBT, DNN, DNN-AE, TabNet, FTT, NPT, Pass, Noise, Sample, CutMix, RFC, WCR, SAINT)
# 2. Generate predictions on the test set
# 3. Compute F1 score for each model
# 4. Rank models by F1 score (1 = best)
# 5. Average ranks across all 28 datasets
# 6. Resolve ties using lower standard deviation of F1 scores
# 7. Apply Wilcoxon-signed rank test for pairwise statistical significance
```

## Common pitfalls

- Several deep models (NPT, TabNet) run out of memory on specific datasets, requiring exclusion from rank averages for those cases.
- Rank ties are broken by comparing the standard deviation of F1 scores across runs, not just mean scores.
- Statistical significance is assessed via Wilcoxon-signed rank test, not just raw F1 differences.

## Evidence (verbatim from paper)

> CutMix yields the highest F1 scores on eight out of 28 data sets. Among these eight data sets, hard and easy-to-classify datasets are evenly split. RFC also shows the best F1 scores on eight other data sets, where hard and easy datasets are in a six-to-two ratio. The average rank (AvgRank) in Table 6 across 28 data sets reveals that CutMix (AvgRank: 6.75) and RFC (AvgRank: 6.96) are superior to traditional data corruption methods, including adding noise (Noise, AvgRank: 8.89) and sampling from distribution (Sample, AvgRank: 7.11).

## Citation

```bibtex
@misc{rabbani2024attention,
  title={Attention versus Contrastive Learning of Tabular Data -- A Data-centric Benchmarking},
  author={Rabbani et al. (2024)},
  year={2024},
  note={arXiv:2401.04266}
}
```

- arXiv: 2401.04266