Dsf Gan Utility Eval

dsf-gan-utility-eval

DSF-GAN: DownStream Feedback Generative Adversarial Network — Perets et al. (2024) (arXiv:2403.18267, 2024)

What this evaluates

Evaluates the predictive utility of synthetic tabular data generated by a GAN. It measures how well a downstream classifier or regressor trained on the synthetic samples performs when evaluated on a strictly held-out real validation set.

Datasets

• Two distinct tabular datasets (names in Appendix A) — total ?; splits: train (-1), test (-1)

Metrics

• model performance (primary) — range: other
  • Accuracy for classification tasks or RMSE/MAE for regression tasks, computed by training a logistic or linear model on synthetic data and evaluating it on a held-out real validation set.

Input / output format

Input: Real tabular dataset for GAN training and a set-aside real validation set for evaluation.

Output: Synthetic tabular samples generated by the trained GAN, used to train a downstream model.

Scoring recipe

# Train GAN for N epochs with downstream feedback loss
# Sample n synthetic samples from trained GAN
synthetic_X, synthetic_y = G.sample(n)
# Train downstream model on synthetic data
model = LogisticRegression() if classification else LinearRegression()
model.fit(synthetic_X, synthetic_y)
# Evaluate on held-out real validation set
val_X, val_y = get_held_out_real_validation_set()
metric_value = model.score(val_X, val_y)

Common pitfalls

• Using synthetic data for validation instead of a strictly held-out real validation set.
• Ambiguity in metric selection: classification tasks require accuracy/F1 while regression requires RMSE/MAE, but the paper only states 'model performance'.
• Failing to exclude the validation set from the GAN's training data, leading to data leakage.

Evidence (verbatim from paper)

Post-training, we sampled $n$ samples from the trained model, and used it as a training set for a regression or classification model, we evaluated the model performance using a set-aside validation set comprised of real samples which were excluded from the GAN training.

Citation

@misc{perets2024dsfgan,
  title={DSF-GAN: DownStream Feedback Generative Adversarial Network},
  author={Perets et al. (2024)},
  year={2024},
  note={arXiv:2403.18267}
}

• arXiv: 2403.18267