Synthetic Data Skill

Synthetic Data Skill

Purpose: generate synthetic datasets for credit-card marketing, risk, and lifecycle experiments. This skill produces canonical tables under data/synthetic including customers.csv, campaigns.csv, presentments.csv, email_events.csv, mail_deliveries.csv, applications.csv, and branch_visits.csv.

Usage:

• Run the main generator:

python3 synthetic-data-skill/scripts/generate_synthetic_data.py

• Summarize campaign-level metrics:

python3 synthetic-data-skill/scripts/generate_synthetic_summary.py

• Render static charts to data/output:

python3 synthetic-data-skill/scripts/render_dashboard_static.py

Notes:

• The generated data is intentionally reusable across other feature-engineering skills (credit card risk, lifecycle, marketing response). Keep this skill isolated to avoid coupling data generation changes with downstream feature code.
• Domain guardrails for feature creation (schema, leakage rules) are maintained in the feature-engineering skill: see customer-marketing-response/references/schema.md and customer-marketing-response/references/leakage.md for guidance on building leakage-safe features.

Domain facts and campaign behavior are documented in: synthetic-data-skill/references/domain.md (campaign counts, per-customer targeting assumptions, product mappings). Use that file as the canonical domain source for generation parameters.

Comprehensive generation rules and guardrails

This section documents the exact data semantics, distributions, constraints, and checks used to produce the CSV artifacts. It is intended to be the canonical, human-readable specification for reproducing the synthetic datasets without relying on the Python implementation.

General principles

• Deterministic reproducibility: every randomized step must reference a named RNG seed. Default seeds used in this repo are documented per-table; to reproduce exactly, use the same seeds and RNG algorithm (NumPy PCG64/Generator).
• Time window baseline: events are generated with timestamps in the 12-month window ending at now = 2026-01-01 unless otherwise noted. Campaign sent_date values are sampled inside 2025 (2025-01-01 → 2025-12-31).
• Segment separation: customers are assigned exactly one segment value in {Customer,Prospect} and downstream sampling of campaign recipients respects campaign type (Customer vs Prospect).
• Channel semantics: campaigns use canonical channel values: email (digital email campaigns) and physical (postal mail). email_events must reference only email campaigns; mail_deliveries must reference only physical campaigns. presentments (portal/app placements) may include exposures for both email and physical campaigns but are sampled with a bias toward email campaigns.

Files produced (canonical list)

• customers.csv — customer master (columns, types, distributions below)
• campaigns.csv — campaign metadata (one row per campaign)
• presentments.csv — portal/app placement exposures (one row per placement)
• email_events.csv — email-level events (delivered/open/click)
• mail_deliveries.csv — physical mail delivery attempts
• applications.csv — application start events (used as response labels)
• branch_visits.csv — branch visit logs (optional auxiliary signal)

Per-table specification

1. customers.csv

   • Columns & types:
     • customer_id (string): format c{NNNNN} zero-padded, unique.
     • dob (date): birth date (used to derive age distribution ~18–75).
     • signup_date (date): account open / signup date in [2018-01-01, 2025-01-01).
     • acquisition_channel (categorical): {organic, email, branch, partner} with default probabilities {0.5, 0.25, 0.15, 0.1}.
     • segment (categorical): {Customer, Prospect} with default probabilities {0.85, 0.15}.
     • owns_credit_card (binary): Bernoulli(p=0.2).
     • owns_debit_card (binary): Bernoulli(p=0.7).
     • owns_home_loan (binary): Bernoulli(p=0.05).
     • owns_personal_loan (binary): Bernoulli(p=0.05).
   • Cardinality: default 50,000 rows (n_customers=50000).

2. campaigns.csv

   • Columns & types:
     • campaign_id (string): format cmp{NNNN} unique.
     • campaign_name (string): free text identifier.
     • type (categorical): {Prospect, Customer} — selection target segment.
     • channel (categorical): {email, physical}.
     • sent_date (date): event date in 2025.
   • Cardinality: default 50 campaigns (n_campaigns=50).
   • Channel distribution: sampled uniformly by default. You can enforce a fixed split (e.g., 70% email / 30% physical) when desired.

3. presentments.csv (portal/app exposures)

   • Columns & types:
     • presentment_id (string): format p{NNNNNNN} unique.
     • customer_id (string): must refer to customers.customer_id and respect segment according to campaign.type when campaign is associated.
     • session_id (string)
     • placement (categorical): {account_summary,account_details,carousel,banner,splash} with default probabilities {0.2,0.2,0.3,0.2,0.1}.
     • placement_type (string): derived label (e.g., carousel,banner,splash).
     • product_shown (categorical): {card,loan,deposit,other} with default probabilities {0.4,0.2,0.2,0.2}.
     • campaign_id (string): may be assigned to either an email or physical campaign (see channel bias rules below); must exist in campaigns.csv.
     • click_flag (binary): Bernoulli(p=0.1) by default.
     • timestamp (datetime): sampled in the 12-month baseline window up to now.
   • Cardinality: default ~300,000 rows (n_presentments=300000).
   • Sampling & bias rules:
     • Presentments are sampled from the pool of campaigns with a weight vector that biases email campaigns higher (default weight email=0.7, physical=0.3). This ensures portal exposures primarily reflect digital campaigns while allowing physical campaigns to surface in the portal/app when the bank also promotes postal campaigns digitally.
     • When campaign.type == 'Customer', presentment customer_id should be drawn exclusively from customers with segment == 'Customer' (and similarly for Prospect).
     • Deduplication guidance: session-level de-duplication should later be applied in feature code (e.g., drop duplicate presentments per (session_id, placement) if desired).

4. email_events.csv

   • Columns & types:
     • message_id (string)
     • customer_id (string)
     • event_type (categorical): {delivered,open,click} with default probabilities {0.6,0.3,0.1}.
     • timestamp (datetime)
     • campaign_id (string): MUST reference a campaign where channel == 'email'.
   • Cardinality: default ~200,000 rows (n_events=200000).
   • Sampling rules:
     • Sample from campaigns with channel == 'email' only. When linking to customers, ensure campaign type mapping (Customer/Prospect) is respected.

5. mail_deliveries.csv

   • Columns & types:
     • delivery_id (string)
     • campaign_id (string): MUST reference a campaign where channel == 'physical'.
     • customer_id (string)
     • timestamp (datetime): near campaign sent_date (default offset 0–7 days).
     • status (categorical): {delivered,undelivered} with default p_delivered=0.98.
   • Sampling rules:
     • For each physical campaign, sample a fraction of the eligible target pool (default 5% of the segment pool per campaign) without replacement. This models postal targeting where only a subset receives physical mail.

6. applications.csv

   • Columns & types:
     • application_id (string)
     • customer_id (string)
     • product (string): e.g., credit_card
     • event_type (string): e.g., start
     • timestamp (datetime)
   • Sampling rules:
     • Applications are generated from presentments where product_shown == 'card' at a base apply_rate (default 2%). Sample presentments and assign application timestamps offset from the presentment by 1–30 days.

7. branch_visits.csv

   • Columns & types: visit_id, customer_id, visit_date, branch_id, purpose.
   • Cardinality: default ~30,000 rows.

Distribution & tunable parameters (summary)

• n_customers (default 50000)
• n_campaigns (default 50)
• n_presentments (default 300000)
• n_email_events (default 200000)
• presentment_channel_weights (email=0.7, physical=0.3)
• apply_rate (default 0.02)
• mail_recipient_fraction (default 0.05 per physical campaign)

Reproducibility & audit checks

• Seed all RNGs with named seeds (document seeds in changelog). Using NumPy's default RNG (np.random.default_rng(seed)) preserves reproducibility across runs when the same seed and NumPy version are used.
• Validation checks to run after generation:
  • All campaign_id values referenced by presentments, email_events, and mail_deliveries exist in campaigns.csv.
  • email_events.campaign_id references only campaigns where channel == 'email'.
  • mail_deliveries.campaign_id references only campaigns where channel == 'physical'.
  • presentments.customer_id values exist in customers.customer_id and respect segment when campaign.type is set.
  • No timestamps are beyond the baseline now (default 2026-01-01).
  • Cardinalities match expected ranges (raise warning if off by >20%).

Guardrails and domain constraints (quadrails)

• Channel purity: the generator must never create email_events that reference non-email campaigns or mail_deliveries that reference non-physical campaigns. Violations should fail the generation run.
• Segment fidelity: if a campaign has type == 'Customer', at least 95% of its direct exposures (presentments or email deliveries) should target customers (not prospects). This allows a small amount of cross-target noise (<5%) optionally configurable.
• No implicit joins by name: all foreign keys must use explicit customer_id and campaign_id columns; do not infer linkages via names or message ids.
• Privacy safety: generated customer_id values must be synthetic (no real identifiers) and the distributions should not be plausible to match real individuals. Avoid using real-world PII templates.
• Change control: when altering distributions or defaults (e.g., changing apply_rate), increment the synthetic-data-skill version and record the change in CHANGELOG.md with rationale.

Human-readable reproduction steps (no code required)

1. Create customers.csv with N rows. Assign segment values such that ~85% are Customer and ~15% Prospect. Sample dob to produce ages roughly between 18 and 75. Set product ownership columns using Bernoulli draws at the probabilities above.
2. Create campaigns.csv with M campaigns scattered across 2025. Assign type (Customer/Prospect) and channel (email/physical).
3. For presentments: decide a target total (e.g., 300k). For each presentment draw:
   • sample a campaign according to weights (email 0.7, physical 0.3);
   • sample a customer from the campaign's target segment pool;
   • sample placement, product_shown, click flag, and timestamp from the described distributions.
4. For email events: sample message events only for email campaigns. Map each event to a customer from the campaign's target segment, assign event_type per distribution, and timestamp in baseline window.
5. For mail deliveries: for each physical campaign, sample ~5% of the segment population without replacement and assign a delivery timestamp near the campaign sent_date.
6. For applications: sample from card presentments at the apply_rate (2%) and set application timestamps 1–30 days after the originating presentment.

Suggested post-generation artifact files

• CHANGELOG.md — record versioned changes to distributions, seeds, and generation logic.
• VALIDATION.md — a checklist of the validation assertions above and their expected pass tolerances.
• specs/ — machine-readable table specs (CSV/JSON) listing column names, types, allowed values, and example rows. These specs make it easy to reproduce in SQL or spreadsheet tools.

Usage notes

• Use synthetic-data-skill to produce canonical, shareable data for experiments. Keep feature-engineering guardrails (time truncation, leakage checks) inside the feature skill(s) — they encode modeling constraints.
• If you need a different scenario (e.g., seasonal campaigns, higher apply rate, or a larger/smaller customer base), change the parameters in the table above and record the change in CHANGELOG.md.

Contact

• For questions about distributions or to propose a new synthetic scenario, edit an issue in the repo or contact the data owner recorded in the project README.