Struct Bench Eval

struct-bench-eval

Struct-Bench: A Benchmark for Differentially Private Structured Text Generation — Wang et al. (2025) (arXiv:2509.10696, 2025)

What this evaluates

This benchmark evaluates the quality of differentially private synthetic structured text generation by measuring how well synthetic datasets preserve the syntactic structure, semantic dependencies, and attribute distributions of real data, alongside downstream task utility.

Datasets

• ShareGPT — total ?; splits: test (-1)

Metrics

• CFG Pass Rate (CFG-PR) (primary) — range: [0, 1]
  • Measures the fraction of samples in the synthetic dataset that parse correctly under the user-provided context-free grammar.
• Key Node Dependency (KND) — range: other
  • Calculates the distributional distance (e.g., Wasserstein-2) between the dependency distributions of key node pairs in the real and synthetic datasets.
• Attribute Match (AM) — range: [0, 1]
  • Measures the distributional distance of sample-level or node-level attributes between real and synthetic datasets, using Wasserstein-2 for numeric attributes or total variation distance for categorical attributes.
• KNN-Precision — range: [0, 1]
  • Calculates the proportion of synthetic samples whose embedding distance to a real sample is smaller than the sample's k-th nearest neighbor within the synthetic dataset.
• KNN-Recall — range: [0, 1]
  • Calculates the proportion of real samples whose embedding distance to a synthetic sample is smaller than the sample's k-th nearest neighbor within the real dataset.
• Downstream Accuracy (Acc) — range: [0, 1]
  • Prediction accuracy on a held-out test set from the real data after training a model on the synthetic dataset, following the TSTR framework.

Input / output format

Input: Real dataset D (string/JSON), synthetic dataset D', a context-free grammar (CFG) defining structural constraints, and an optional set of key node pairs.

Output: Numerical scores for CFG Pass Rate, Key Node Dependency distance, Attribute Match distance, KNN-Precision, KNN-Recall, and downstream prediction accuracy.

Scoring recipe

cfg_pass = sum(1 for s in D_syn if parse(s, cfg)) / len(D_syn)
real_deps = [cosine_sim(embed(n1), embed(n2)) for n1, n2 in key_pairs(D_real)]
syn_deps = [cosine_sim(embed(n1), embed(n2)) for n1, n2 in key_pairs(D_syn)]
knd = wasserstein_2(real_deps, syn_deps)
real_attrs = extract_attrs(D_real, attr_fn)
syn_attrs = extract_attrs(D_syn, attr_fn)
am = wasserstein_2(real_attrs, syn_attrs) if numeric else tv_distance(real_attrs, syn_attrs)
knn_prec = compute_knn_precision(D_syn, D_real, k)
knn_rec = compute_knn_recall(D_real, D_syn, k)
model = train(D_syn, labels_syn)
acc = evaluate(model, D_real_test, labels_real_test)

Common pitfalls

• CFG specification is performed manually once per dataset and may be error-prone or require significant domain knowledge.
• KND only measures pairwise dependencies between key nodes, ignoring higher-order relationships which are computationally intensive.
• Attribute Match distance metric must be selected based on data type: Wasserstein-2 for numeric attributes and total variation distance for categorical attributes.
• The TSTR framework is used for fair algorithm comparison, but differs from TRTR which evaluates self-contained utility.

Evidence (verbatim from paper)

We report three types of metrics: structural, non-structural, and downstream task accuracy. (1) CFG Pass Rate (CFG-PR): This measures the fraction of samples in the synthetic dataset D′ that parse correctly under the CFG. (2) Key Node Dependency (KND): This metric measures the semantic dependencies between “key node pairs”, which are pairs of nodes believed to have a meaningful relation; for example, in a question-and-answer dataset, we would expect that each associated question and answer pair should have a strong correlation. Programmatically, users specify pairs of key nodes with Tregex, a tool for matching regular expressions on trees. Typically, we can measure the dependencies by cosine similarity of the node embeddings, while one can also adopt LLM as a judge or other dependency functions defined by the users. For a dataset, we can construct a distribution of dependencies of node pairs in the same pattern. To evaluate the similarity of the node dependencies captured by the private and synthetic dataset, we then calculate the distributional distance, e.g., Wasserstein-2 distance, between the private and synthetic dependency distributions. (3) Attribute Match (AM): This metric

Citation

@misc{wang2025structbench,
  title={Struct-Bench: A Benchmark for Differentially Private Structured Text Generation},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2509.10696}
}

• arXiv: 2509.10696