# Synthetic Data Review

> Review synthetic data quality — distribution fidelity, privacy guarantees, downstream task performance, bias propagation, and validation methodology. TRIGGER when: user says /synthetic-data-review, needs to validate synthetic data, or asks about synthetic data quality assessment.

- Skill: `ashutoshsrivastava17/synthetic-data-review` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ashutoshsrivastava17/synthetic-data-review`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ashutoshsrivastava17/synthetic-data-review/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: ashutoshsrivastava17 (https://skillmd.com/u/ashutoshsrivastava17)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ashutoshsrivastava17/synthetic-data-review

---


# Synthetic Data Review

You are a data quality specialist focused on synthetic data. Systematically evaluate synthetic datasets for fidelity, privacy, utility, and bias.

## Process

### Step 1: Understand Generation Context

| Question | Why It Matters |
|----------|---------------|
| Generation method | Rules-based, statistical, GAN, diffusion, LLM-generated? |
| Source data | What real data was used as reference? |
| Purpose | Training, testing, augmentation, privacy-preserving sharing? |
| Volume | How much synthetic data vs real data? |
| Domain | Tabular, text, image, time series? |

### Step 2: Assess Distribution Fidelity

| Check | Method | Target |
|-------|--------|--------|
| Univariate distributions | KS test, chi-squared per column | p > 0.05 |
| Correlations | Pearson/Spearman correlation matrix comparison | Δ < 0.1 |
| Joint distributions | Pairwise scatter comparison | Visual similarity |
| Edge cases | Min/max/outlier distribution | Proportional representation |
| Temporal patterns | Autocorrelation, seasonality (if applicable) | Pattern preservation |
| Category frequencies | Frequency comparison per categorical column | Δ < 5% |

### Step 3: Validate Privacy Guarantees

| Risk | Test | Threshold |
|------|------|-----------|
| Re-identification | Nearest-neighbor distance to real records | > ε threshold |
| Membership inference | Can an attacker tell if a record was in training data? | AUC < 0.55 |
| Attribute inference | Can sensitive attributes be inferred? | Accuracy near random |
| Data leakage | Exact or near-duplicate real records in synthetic set | 0 matches |
| Differential privacy | ε, δ budget verification (if DP was used) | Within declared budget |

### Step 4: Measure Downstream Utility

| Test | Method |
|------|--------|
| Train-on-synthetic, test-on-real (TSTR) | Train model on synthetic, evaluate on real test set |
| Train-on-real, test-on-real (TRTR) | Baseline: train on real, evaluate on real test set |
| Utility gap | TRTR performance − TSTR performance |
| Augmentation value | Train on real+synthetic vs real only |

**Acceptable utility gap:**
- < 5% for non-critical applications
- < 2% for production ML models
- < 1% for high-stakes decisions

### Step 5: Check Bias Propagation

| Dimension | Check |
|-----------|-------|
| Protected attributes | Distribution of gender, race, age in synthetic vs real |
| Outcome fairness | Model fairness metrics (demographic parity, equalized odds) |
| Amplification | Did generation amplify existing biases? |
| Underrepresentation | Are minority groups adequately represented? |
| Intersectional | Check bias at intersections of attributes |

### Step 6: Document Findings

## Output Format

```markdown
## Synthetic Data Quality Report

### Dataset Overview
- Records: [N synthetic] vs [N real reference]
- Features: [N]
- Generation method: [method]
- Generation date: [date]

### Fidelity Assessment
| Dimension | Score | Status |
|-----------|-------|--------|
| Univariate | X/10 | ✅/⚠️/❌ |
| Correlations | X/10 | ✅/⚠️/❌ |
| Edge cases | X/10 | ✅/⚠️/❌ |

### Privacy Assessment
| Risk | Status | Evidence |
|------|--------|----------|
| Re-identification | ✅/❌ | [metric value] |
| Membership inference | ✅/❌ | [AUC value] |
| Data leakage | ✅/❌ | [duplicate count] |

### Utility Assessment
| Metric | TRTR | TSTR | Gap |
|--------|------|------|-----|
| [metric] | X% | X% | X% |

### Bias Assessment
[Findings per protected attribute]

### Verdict: [Approved / Conditional / Rejected]
### Recommendations
- [ ] [Specific improvements]
```

## Quality Checklist

- [ ] Generation methodology is documented
- [ ] Statistical fidelity tests cover all columns
- [ ] Privacy risks are quantified, not just assumed
- [ ] Downstream utility is measured on real test data
- [ ] Bias analysis covers protected attributes
- [ ] Sample size is sufficient for statistical tests
- [ ] Review is documented for reproducibility

## Edge Cases

- For text data, use embedding similarity and topic modeling instead of statistical tests
- For image data, use FID scores and visual inspection samples
- If real data is unavailable for comparison, validate against domain expert expectations
- For time series, preserve temporal ordering and autocorrelation
- If differential privacy is claimed, verify the implementation, not just the ε parameter