Synthetic Data Eficacy Eval

synthetic-data-eficacy-eval

Efficacy of Synthetic Data as a Benchmark — Maheshwari et al. (2024) (arXiv:2409.11968, 2024)

What this evaluates

Evaluates whether synthetic data generated by LLMs can effectively substitute real data for benchmarking NLP models, measuring both absolute performance alignment and relative ranking preservation across tasks. Additionally quantifies the self-bias of LLMs when they generate data and subsequently solve the same tasks.

Datasets

• Headlines — total ?; splits: test (-1); repo https://github.com/Diabolocom-Research/SG4NLP
• Tweet-News — total ?; splits: test (-1); repo https://github.com/Diabolocom-Research/SG4NLP
• CrossNER-Literature — total ?; splits: test (-1); repo https://github.com/Diabolocom-Research/SG4NLP
• CrossNER-Politics — total ?; splits: test (-1); repo https://github.com/Diabolocom-Research/SG4NLP
• SNIPS — total ?; splits: test (-1); repo https://github.com/Diabolocom-Research/SG4NLP
• ATIS — total ?; splits: test (-1); repo https://github.com/Diabolocom-Research/SG4NLP

Metrics

• MSPD (primary) — range: other
  • Mean Square Performance Difference. Computes the average squared difference between the performance of each method on the real dataset and the synthetic dataset: sum((m_real - m_syn)^2) / n_methods.
• SRCC — range: [-1, 1]
  • Spearman Rank Correlation Coefficient. Measures the monotonic relationship between the rankings of methods on the real dataset versus the synthetic dataset.
• Bias Factor — range: [-1, 1]
  • Normalized performance of an LLM on its own generated data minus the average normalized performance across data generated by other LLMs. Normalization subtracts the mean performance of all LLMs on that specific synthetic dataset.

Input / output format

Input: Prompt containing task description, few-shot examples from real data, generation guidelines, and output description for data generation; or task description, output description, and input data point for zero-shot task solving.

Output: Synthetic dataset instances (text/labels) for generation; predicted labels/scores for task solving.

Scoring recipe

def compute_metrics(real_data, syn_data, methods):
    perf_real = {m: metric(real_data, m) for m in methods}
    perf_syn = {m: metric(syn_data, m) for m in methods}
    mspd = mean((perf_real[m] - perf_syn[m])**2 for m in methods)
    srcc = spearman_rank(perf_real.values(), perf_syn.values())
    # Bias Factor
    norm_own = perf_syn[own_llm] - mean(perf_syn.values())
    norm_others = mean(perf_syn[m] for m in methods if m != own_llm)
    bias = norm_own - norm_others
    return mspd, srcc, bias

Common pitfalls

• High MSPD does not invalidate the synthetic dataset; it only indicates poor absolute performance alignment, while SRCC may still be high, preserving useful relative rankings.
• Bias factor calculation requires normalizing performance scores first; raw scores >1 must be scaled to [0,1] before computing bias to avoid skewed results.
• The protocol assumes the same LLM is used for both data generation and task solving, which can artificially inflate performance on its own generated data.

Evidence (verbatim from paper)

To capture the absolute performance difference, we propose mean squared performance difference (MSPD): ... To capture this relative performance, we rely on Spearman’s rank correlation measure between the two sets ... Finally, the bias factor for LLM K for dataset D is:

Citation

@misc{maheshwari2024efficacy,
  title={Efficacy of Synthetic Data as a Benchmark},
  author={Maheshwari et al. (2024)},
  year={2024},
  note={arXiv:2409.11968}
}

• arXiv: 2409.11968