Synthony Selection Eval

synthony-selection-eval

SYNTHONY: A Stress-Aware, Intent-Conditioned Agent for Deep Tabular Generative Models Selection — Son et al. (2026) (arXiv:2604.00293, 2026)

What this evaluates

Probes the ability of an agent to select the optimal tabular data synthesizer for a given dataset and objective (privacy, fidelity, or utility) based on dataset stress profiles and a capability registry. It evaluates whether stress-aware, intent-conditioned matching outperforms heuristics, zero-shot LLMs, and meta-learning baselines in ranking generative models.

Datasets

• OpenML Tabular Benchmark (Abalone, Bean, IndianLiverPatient, Obesity, faults, insurance, wilt) — total ?; splits: train (12), test (9)

Metrics

• Top-3 Accuracy (primary) — range: [0, 1]
  • Fraction of dataset-intent pairs where the oracle-best model appears in the agent’s top-3 recommendations.
• Spearman Rank Correlation (primary) — range: [-1, 1]
  • Measures agreement between predicted and oracle full rankings across all 10 models, rewarding correct ordering across the entire ranking.
• Top-1 Accuracy — range: [0, 1]
  • Fraction of pairs where the selector’s top pick matches the oracle-best model.
• NDCG — range: [0, 1]
  • Normalized Discounted Cumulative Gain evaluating the full predicted ranking with position-weighted relevance.

Input / output format

Input: Dataset stress profile (4 interpretable dimensions derived from meta-features), selection intent (privacy, fidelity, or utility), and a candidate pool of 10 synthesizers.

Output: A ranked list of synthesizers (top-10) for the specified dataset-intent pair.

Scoring recipe

def compute_metrics(predictions, gold_rankings):
    top1, top3, spearman_vals, ndcg_vals = 0, 0, [], []
    for key in predictions:
        pred, gold = predictions[key], gold_rankings[key]
        if pred[0] == gold[0]: top1 += 1
        if gold[0] in pred[:3]: top3 += 1
        spearman_vals.append(spearman_corr(pred, gold))
        ndcg_vals.append(ndcg_at_k(pred, gold, k=len(gold)))
    n = len(predictions)
    return {'top1_accuracy': top1/n, 'top3_accuracy': top3/n,
            'spearman': mean(spearman_vals), 'ndcg': mean(ndcg_vals)}

Common pitfalls

• The kNN baseline interpolates from oracle rankings of training datasets, making it an oracle-informed reference point rather than a deployable zero-knowledge method; direct comparison to zero-knowledge baselines is structurally unfair.
• Top-1 accuracy is highly sensitive to the small test set size (9 pairs), where each correct prediction shifts accuracy by ~11.1 percentage points, so minor differences should be interpreted cautiously.
• Privacy intent exhibits high model diversity (5 distinct winners across 7 datasets), making it the hardest objective; selectors that default to majority-class models (e.g., CART) will appear strong on fidelity/utility but fail completely on privacy.

Evidence (verbatim from paper)

We assess selection quality with four metrics computed over the 10-model rankings. Our two primary metrics are Top-3 Accuracy, the fraction of dataset-intent pairs where the oracle-best model appears in the agent’s top-3 recommendations (reflecting the realistic deployment scenario of presenting a short list for downstream evaluation), and Spearman Rank Correlation, which measures agreement between predicted and oracle full rankings across all 10 models ($[-1,1]$) and rewards correct ordering across the entire ranking. As secondary metrics we report Top-1 Accuracy, the fraction of pairs where the selector’s top pick matches the oracle-best model (with only 9 test pairs, each correct prediction shifts Top-1 by 11.1 percentage points, so we interpret differences smaller than ${\sim}$0.15 with caution), and NDCG (Normalized Discounted Cumulative Gain), which evaluates the full predicted ranking with position-weighted relevance ($[0,1]$).

Citation

@misc{son2026synthony,
  title={SYNTHONY: A Stress-Aware, Intent-Conditioned Agent for Deep Tabular Generative Models Selection},
  author={Son et al. (2026)},
  year={2026},
  note={arXiv:2604.00293}
}

• arXiv: 2604.00293