Synthetic Tabular Data Benchmark Eval

synthetic-tabular-data-benchmark-eval

Reimagining Synthetic Tabular Data Generation through Data-Centric AI: A Comprehensive Benchmark — Hansen et al. (2023) (arXiv:2310.16981, 2023)

What this evaluates

Evaluates the downstream utility and statistical fidelity of synthetic tabular data generated by various models. It probes whether synthetic data preserves classification accuracy, model selection rankings, feature importance rankings, and distributional similarity compared to real data.

Datasets

• Tabular Classification from Numerical features benchmark suite (filtered) — total ?; splits: train (-1); repo https://github.com/HLasse/data-centric-synthetic-data

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve. Measures the probability that a classifier ranks a random positive instance higher than a random negative instance.
• Spearman's Rank Correlation — range: [-1, 1]
  • Non-parametric measure of rank correlation. Computes the correlation between the rank orders of two variables to assess how well the relative ordering is preserved.
• Inverse KL divergence — range: [0, 1]
  • Reciprocal of the Kullback-Leibler divergence between the real and synthetic data distributions. Used to quantify statistical fidelity, where higher values indicate closer distributional match.

Input / output format

Input: Synthetic tabular datasets generated by conditional generative models (e.g., CTGAN, TVAE, DDPM, NFLOW, Bayesian Networks), optionally subjected to data-centric preprocessing (Cleanlab, Data-IQ, Data Maps) or label noise injection.

Output: Predicted probabilities or class labels for classification tasks, and ranked lists of supervised models or features for selection tasks, compared against ground truth rankings from real data.

Scoring recipe

# 1. Classification (AUROC)
auroc = roc_auc_score(y_true, y_scores)

# 2. Model/Feature Selection (Spearman's Rank Correlation)
rho, _ = spearmanr(ranking_real, ranking_synthetic)

# 3. Statistical Fidelity (Inverse KL Divergence)
# Distributions are typically estimated via histograms or KDE
kl = kl_divergence(p_real, p_synthetic)
fidelity = 1.0 / kl

Common pitfalls

• Assuming high statistical fidelity (low KL divergence) directly translates to high downstream task utility.
• Ignoring data-centric preprocessing steps (e.g., Cleanlab) which can drastically alter the proportion of easy vs hard samples and skew downstream rankings.
• Evaluating only a single supervised model (e.g., XGBoost) for all metrics without assessing model selection across diverse architectures.

Evidence (verbatim from paper)

The classification performance is evaluated in terms of the area under the receiver operating characteristic curve (AUROC), model selection performance as Spearman's Rank Correlation between the ranking of the supervised classification models trained on the original data and the supervised classification models trained on the synthetic data, and feature selection performance as Spearman's Rank Correlation between the ranking of features in an xgboost model trained on the original data and an xgboost model trained on the synthetic data.

Citation

@misc{hansen2023reimagining,
  title={Reimagining Synthetic Tabular Data Generation through Data-Centric AI: A Comprehensive Benchmark},
  author={Hansen et al. (2023)},
  year={2023},
  note={arXiv:2310.16981}
}

• arXiv: 2310.16981