Tabular Cleaning Eval

tabular-cleaning-eval

Prior-Aligned Data Cleaning for Tabular Foundation Models — Berti-Equille et al. (2026) (arXiv:2604.25154, 2026)

What this evaluates

Evaluates automated tabular data cleaning pipelines by measuring downstream classification accuracy and calibration when processed by a Tabular Foundation Model (TabPFN v2). It probes whether cleaning strategies can effectively align dirty data distributions with the model's learned prior to improve predictive performance.

Datasets

• OpenML CC18 Benchmark Suite (D1–D10) — total ?; splits: train (-1), test (-1); repo https://github.com/LaureBerti/Learn2Clean

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted class labels on the held-out test split.
• ECE — range: [0, 1]
  • Expected Calibration Error computed over 10 equal-width confidence bins on the softmax probability of the predicted class. Calculated as the weighted average of the absolute difference between bin accuracy and bin confidence.

Input / output format

Input: Tabular dataset (rows × features) containing natural or synthetically injected errors (MCAR, MAR, outliers, duplicates) and missing values.

Output: Softmax class probabilities and predicted class label from TabPFN v2.

Scoring recipe

correct = sum(pred == gold for pred, gold in zip(predictions, gold_labels))
accuracy = correct / len(gold_labels)

bins = 10
bin_width = 1.0 / bins
ece = 0.0
for i in range(bins):
    low, high = i * bin_width, (i + 1) * bin_width
    mask = (confidence >= low) & (confidence < high)
    if mask.sum() > 0:
        bin_acc = (pred[mask] == gold[mask]).mean()
        bin_conf = confidence[mask].mean()
        ece += (mask.sum() / len(gold)) * abs(bin_acc - bin_conf)

Common pitfalls

• Using TabPFN v1 instead of v2, as v1 lacks the unconditional z-normalization, power transform, and missing-value flags that critically affect sensitivity to upstream data quality.
• Relying solely on accuracy for class-imbalanced datasets (e.g., Blood Transfusion, Adult, Bank Marketing), where it may understate minority-class benefits; ECE should be consulted for calibration insights.
• Assuming high reward function scores correlate with high downstream accuracy, as trivial-collapse rewards (e.g., completeness retention) saturate at ~1.0 while yielding poor predictive performance.

Evidence (verbatim from paper)

All cleaning policies are finally evaluated by TabPFN v2 accuracy and ECE on a 20% held-out test split (stratified, seed=42). ECE is computed with 10 equal-width confidence bins on the softmax probability of the predicted class. Accuracy is the primary metric for three reasons: (i) it is the standard reported by TabPFN v2’s own benchmark suite*(Hollmann et al., 2025)* and the OpenML repository for these tasks, enabling direct comparison with published baselines; (ii) seven of the ten datasets have near-balanced class distributions, where accuracy and AUROC are empirically tightly correlated; and (iii) since all methods are evaluated under identical conditions, the ranking of cleaning strategies is robust to the choice of aggregation metric when the pipeline affects the data distribution uniformly across classes—which prior-alignment cleaning does by construction.

Citation

@misc{berti2026prioraligned,
  title={Prior-Aligned Data Cleaning for Tabular Foundation Models},
  author={Berti-Equille et al. (2026)},
  year={2026},
  note={arXiv:2604.25154}
}

• arXiv: 2604.25154