# Synthony Selection Eval

> Probes the ability of an agent to select the optimal tabular data synthesizer for a given dataset and objective (privacy, fidelity, or utility) based on dataset stress profiles and a capability registry. It evaluates whether stress-aware, intent-conditioned matching outperforms heuristics, zero-shot LLMs, and meta-learning baselines in ranking generative models. Use when the user wants to benchmark on OpenML Tabular Benchmark (Abalone, Bean, IndianLiverPatient, Obesity, faults, insurance, wilt), or asks about evaluating this task. Reports Top-3 Accuracy, Spearman Rank Correlation.

- Skill: `qhjqhj00/synthony-selection-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/synthony-selection-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/synthony-selection-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/synthony-selection-eval

---


# synthony-selection-eval

> SYNTHONY: A Stress-Aware, Intent-Conditioned Agent for Deep Tabular Generative Models Selection — Son et al. (2026) (arXiv:2604.00293, 2026)

## What this evaluates

Probes the ability of an agent to select the optimal tabular data synthesizer for a given dataset and objective (privacy, fidelity, or utility) based on dataset stress profiles and a capability registry. It evaluates whether stress-aware, intent-conditioned matching outperforms heuristics, zero-shot LLMs, and meta-learning baselines in ranking generative models.

## Datasets

- **OpenML Tabular Benchmark (Abalone, Bean, IndianLiverPatient, Obesity, faults, insurance, wilt)** — total ?; splits: train (12), test (9)

## Metrics

- `Top-3 Accuracy` **(primary)** — range: [0, 1]
  - Fraction of dataset-intent pairs where the oracle-best model appears in the agent’s top-3 recommendations.
- `Spearman Rank Correlation` **(primary)** — range: [-1, 1]
  - Measures agreement between predicted and oracle full rankings across all 10 models, rewarding correct ordering across the entire ranking.
- `Top-1 Accuracy` — range: [0, 1]
  - Fraction of pairs where the selector’s top pick matches the oracle-best model.
- `NDCG` — range: [0, 1]
  - Normalized Discounted Cumulative Gain evaluating the full predicted ranking with position-weighted relevance.

## Input / output format

**Input**: Dataset stress profile (4 interpretable dimensions derived from meta-features), selection intent (privacy, fidelity, or utility), and a candidate pool of 10 synthesizers.

**Output**: A ranked list of synthesizers (top-10) for the specified dataset-intent pair.

## Scoring recipe

```python
def compute_metrics(predictions, gold_rankings):
    top1, top3, spearman_vals, ndcg_vals = 0, 0, [], []
    for key in predictions:
        pred, gold = predictions[key], gold_rankings[key]
        if pred[0] == gold[0]: top1 += 1
        if gold[0] in pred[:3]: top3 += 1
        spearman_vals.append(spearman_corr(pred, gold))
        ndcg_vals.append(ndcg_at_k(pred, gold, k=len(gold)))
    n = len(predictions)
    return {'top1_accuracy': top1/n, 'top3_accuracy': top3/n,
            'spearman': mean(spearman_vals), 'ndcg': mean(ndcg_vals)}
```

## Common pitfalls

- The kNN baseline interpolates from oracle rankings of training datasets, making it an oracle-informed reference point rather than a deployable zero-knowledge method; direct comparison to zero-knowledge baselines is structurally unfair.
- Top-1 accuracy is highly sensitive to the small test set size (9 pairs), where each correct prediction shifts accuracy by ~11.1 percentage points, so minor differences should be interpreted cautiously.
- Privacy intent exhibits high model diversity (5 distinct winners across 7 datasets), making it the hardest objective; selectors that default to majority-class models (e.g., CART) will appear strong on fidelity/utility but fail completely on privacy.

## Evidence (verbatim from paper)

> We assess selection quality with four metrics computed over the 10-model rankings. Our two primary metrics are Top-3 Accuracy, the fraction of dataset-intent pairs where the oracle-best model appears in the agent’s top-3 recommendations (reflecting the realistic deployment scenario of presenting a short list for downstream evaluation), and Spearman Rank Correlation, which measures agreement between predicted and oracle full rankings across all 10 models ($[-1,1]$) and rewards correct ordering across the entire ranking. As secondary metrics we report Top-1 Accuracy, the fraction of pairs where the selector’s top pick matches the oracle-best model (with only 9 test pairs, each correct prediction shifts Top-1 by 11.1 percentage points, so we interpret differences smaller than ${\sim}$0.15 with caution), and NDCG (Normalized Discounted Cumulative Gain), which evaluates the full predicted ranking with position-weighted relevance ($[0,1]$).

## Citation

```bibtex
@misc{son2026synthony,
  title={SYNTHONY: A Stress-Aware, Intent-Conditioned Agent for Deep Tabular Generative Models Selection},
  author={Son et al. (2026)},
  year={2026},
  note={arXiv:2604.00293}
}
```

- arXiv: 2604.00293