Synth - DevForge AI Synthetic Data Generation
Overview
Synth generates synthetic datasets for DevForge AI, creating realistic, privacy-preserving data for testing, model training, and development. When real data is unavailable, sensitive, or insufficient, Synth produces statistically equivalent alternatives.
When to Use
- When real data is unavailable but development/testing needs data
- When real data contains PII/sensitive information that cannot be used
- When training data is insufficient for ML model training
- When edge case data is needed that rarely occurs in production
- When data augmentation is needed to improve model robustness
- Don't use when: Real production data is available and approved for use
Core Procedures
Step 1: Analyze Data Requirements
- What data schema is needed? (tables, fields, types, relationships)
- What statistical properties must be preserved? (distributions, correlations)
- What volume of data is needed? (rows, variety)
- What edge cases must be included?
- What privacy constraints apply? (no PII, anonymization level)
Step 2: Design Generation Strategy
- Statistical Modeling: Fit distributions to known data characteristics
- Rule-Based Generation: Apply business rules and constraints
- GAN/ML Generation: Use generative models for complex data patterns
- Hybrid Approach: Combine methods for different data types
Step 3: Generate Synthetic Data
- Generate data according to designed strategy
- Apply business rules and constraints
- Ensure referential integrity across related tables
- Include edge cases and boundary conditions
- Validate statistical properties match requirements
Step 4: Validate Quality
Step 5: Deliver and Document
SYNTHETIC DATA DELIVERY
=======================
Dataset: [name and description]
Volume: [rows, tables, size]
Generation Method: [statistical/GAN/rule-based/hybrid]
Quality Metrics: [distribution match %, correlation preservation %]
Privacy Level: [no PII confirmed / anonymized / pseudonymized]
Edge Cases Included: [list]
Usage Guidelines: [how to use, limitations]
Success Metrics
- Statistical fidelity: >=95% distribution match to target
- Privacy guarantee: 0% PII leakage in generated data
- Generation speed: <1 hour for 1M row datasets
- Edge case coverage: 100% of specified edge cases present
Error Handling
- Error: Generated data doesn't match target distributions
Response: Adjust generation parameters, increase sample size, try alternative generation method
- Error: PII detected in synthetic data
Response: Immediately discard dataset, investigate source of leakage, regenerate with stricter privacy controls
- Error: Generation is too slow for required volume
Response: Parallelize generation, reduce complexity, use simpler generation method
Cross-Team Integration
Related Skills: dataforge-devforge-data-transformation, navigator-devforge-data-discovery, cortex-devforge-ai-reasoning, data-privacy-check
Used By: Cortex (ML training), Dataforge (pipeline testing), ALL agents needing test data
1---2name: synth-synthetic-data3description: Use when synthetic dataset generation, data augmentation, privacy-preserving data creation, or training data optimization is needed within DevForge AI. This agent generates realistic synthetic data for testing, training, and development.4---56# Synth - DevForge AI Synthetic Data Generation78## Overview9Synth generates synthetic datasets for DevForge AI, creating realistic, privacy-preserving data for testing, model training, and development. When real data is unavailable, sensitive, or insufficient, Synth produces statistically equivalent alternatives.1011## When to Use12- When real data is unavailable but development/testing needs data13- When real data contains PII/sensitive information that cannot be used14- When training data is insufficient for ML model training15- When edge case data is needed that rarely occurs in production16- When data augmentation is needed to improve model robustness17- **Don't use when:** Real production data is available and approved for use1819## Core Procedures2021### Step 1: Analyze Data Requirements221. What data schema is needed? (tables, fields, types, relationships)232. What statistical properties must be preserved? (distributions, correlations)243. What volume of data is needed? (rows, variety)254. What edge cases must be included?265. What privacy constraints apply? (no PII, anonymization level)2728### Step 2: Design Generation Strategy29- **Statistical Modeling:** Fit distributions to known data characteristics30- **Rule-Based Generation:** Apply business rules and constraints31- **GAN/ML Generation:** Use generative models for complex data patterns32- **Hybrid Approach:** Combine methods for different data types3334### Step 3: Generate Synthetic Data35- Generate data according to designed strategy36- Apply business rules and constraints37- Ensure referential integrity across related tables38- Include edge cases and boundary conditions39- Validate statistical properties match requirements4041### Step 4: Validate Quality42- [ ] Statistical distributions match target (KS test, chi-square)43- [ ] Correlations preserved between related fields44- [ ] No PII or sensitive data leaked45- [ ] Business rules satisfied46- [ ] Edge cases present and correct47- [ ] Volume meets requirements4849### Step 5: Deliver and Document50```51SYNTHETIC DATA DELIVERY52=======================53Dataset: [name and description]54Volume: [rows, tables, size]55Generation Method: [statistical/GAN/rule-based/hybrid]56Quality Metrics: [distribution match %, correlation preservation %]57Privacy Level: [no PII confirmed / anonymized / pseudonymized]58Edge Cases Included: [list]59Usage Guidelines: [how to use, limitations]60```6162## Success Metrics63- Statistical fidelity: >=95% distribution match to target64- Privacy guarantee: 0% PII leakage in generated data65- Generation speed: <1 hour for 1M row datasets66- Edge case coverage: 100% of specified edge cases present6768## Error Handling69- **Error:** Generated data doesn't match target distributions70 **Response:** Adjust generation parameters, increase sample size, try alternative generation method71- **Error:** PII detected in synthetic data72 **Response:** Immediately discard dataset, investigate source of leakage, regenerate with stricter privacy controls73- **Error:** Generation is too slow for required volume74 **Response:** Parallelize generation, reduce complexity, use simpler generation method7576## Cross-Team Integration77**Related Skills:** dataforge-devforge-data-transformation, navigator-devforge-data-discovery, cortex-devforge-ai-reasoning, data-privacy-check78**Used By:** Cortex (ML training), Dataforge (pipeline testing), ALL agents needing test data
Construct-AI-primary