Synthetic Data

Synthetic Data Generation for AI Functions

Generate realistic data for evaluating and optimizing AI functions.

Automated Generation (Recommended)

Use the GENERATE_SYNTHETIC_DATA stored procedure for server-side data generation using Cortex LLM.

When to Use

Load this sub-skill when user intent involves: "generate data", "synthetic data", "test data", "create test cases", "make training data", "pseudo label", or "label my input-only table".

Workflow

Follow these steps sequentially when generating synthetic data.

Step 1: Infer Task Description (Preferred)

IMPORTANT: The GENERATE_SYNTHETIC_DATA procedure requires a task description string, NOT a function name.

When an AI function is already known in context (for example from optimize/evaluate workflows, or when the user named the function in their initial message), do not ask the user to retype intention. Infer task_description in this order:

1. Function COMMENT (intention text)
2. System prompt from function DDL (fallback)

If the user named the function up front and inference succeeds, announce the inferred description and proceed without stopping for confirmation. Otherwise (interactive setup with no function context), present a brief confirmation prompt:

I inferred this task description from your function metadata:
{inferred_task_description}

Use this task description? (y/n) If no, provide an updated description.

Only if inference fails (no comment and no usable prompt), ask the user for a task description:

Describe the task your AI function performs. This guides synthetic test-case generation.

Example: "Classify customer support tickets into categories: billing, technical, account, general. Return the category and priority level."

Store the final value as task_description.

Step 2: Collect Input Columns & Output Schema

Ask the user:

What are the input columns (in function argument order)?
What is the output schema for the function? Provide either:
  - An OUTPUT_SCHEMA (JSON object with "properties"), or
  - A FUNCTION_NAME of an existing structured AI function (schema will be inferred from its response_format)

Example:
- Input columns: CLAIM, DOCUMENTS
- Output schema: {"properties": {"VERDICT": {"type": "string"}, "RATIONALE": {"type": "string"}, "EVIDENCE": {"type": "string"}, "CONFIDENCE": {"type": "number"}}}

Store input_columns. For the SQL call, format as comma-separated, single-quoted values: input_columns_csv = 'CLAIM', 'DOCUMENTS'. Store output_schema as a JSON object, or function_name if inferring from an existing function.

Pass --function-name whenever the target function already exists or any input column is semi-structured (ARRAY, VARIANT, OBJECT). Do not rely on --output-schema alone in those cases — only the function DDL exposes per-parameter types, which the generator needs in order to emit valid JSON values for each semi-structured input (e.g. a JSON array like ["billing","tech"] for ARRAY, a JSON object like {"author":"...","date":"..."} for OBJECT) instead of degenerate comma-separated strings. Skipping --function-name will silently produce malformed data that fails downstream evaluate/optimize.

Step 3: Collect Output Table

Ask the user:

What is the fully qualified table name where synthetic data should be stored?

Example: MY_DB.MY_SCHEMA.SYNTHETIC_DATA

Store as output_table.

Step 4: Collect Model

Load references/model_selection.md and follow its full workflow, biasing toward the strongest available model since synthetic data quality depends heavily on model capability.

Store as model. Default: claude-opus-4-6

Step 5: Execute Generation

Run the generation script directly. It renders the anonymous SPROC, appends the CALL, and executes everything in a single Snowpark session. Always pass every flag — use none for unused optional parameters:

PYTHONPATH=<SKILL_DIRECTORY>/src uv run --project <SKILL_DIRECTORY> python <SKILL_DIRECTORY>/src/run.py synthetic \
    --database {database} \
    --schema {schema} \
    --connection {connection} \
    --task-description "{task_description}" \
    --output-table {output_table} \
    --input-columns {input_columns...} \
    --model {model} \
    --num-examples 50 \
    --source-table none \
    --function-name {function_name or none} \
    --output-schema {output_schema_json or none} \
    --max-source-rows none

The script prints a JSON result to stdout:

{"status": "success", "result": {...}, "output_table": "DB.SCHEMA.SYNTHETIC_DATA"}

Note: --input-columns is required. Output shape must be provided via --output-schema or --function-name (at least one must not be none). Note: Outputs are generated under an outputs key (JSON object) with keys matching the schema properties.

Step 6: Show Results

Display the results to the user, including the row count and output table name so they know the workflow finished:

Synthetic data generation complete! Created {total_generated} examples in {output_table}.

Output table: {output_table}
Total examples generated: {total_generated}

You can view your data with:
  SELECT * FROM {output_table} LIMIT 10;

The table has the following columns:
  - ID: Auto-incrementing identifier
  - Input columns: One VARCHAR column per input specified by `INPUT_COLUMNS`
  - EXPECTED: VARIANT column containing JSON object with keys from the output schema

Note: The EXPECTED column is VARIANT (JSON object), not VARCHAR. The evaluate and optimize
pipelines handle this automatically — they convert both EXPECTED and PREDICTED to strings
internally. No manual casting is needed when passing this table to evaluate/optimize.

To access individual expected values:
  SELECT EXPECTED:key_name FROM {output_table};

If there were batch errors, report them but note that partial data may still be usable.

Step 7: Continue

Ask the user what they want to do next:

What would you like to do next?

1. **View the data** - Preview the generated examples
2. **Evaluate** - Run your AI function against this data
3. **Optimize** - Use this data to optimize your AI function's prompt
4. **Generate more** - Create additional synthetic data
5. **Done** - Finished with data generation

Route based on selection:

• View: Execute SELECT * FROM {output_table} LIMIT 10;
• Evaluate: Load evaluate/SKILL.md
• Optimize: Load optimize/SKILL.md
• Generate more: Return to Step 1
• Done: End workflow

Pseudo-Label Input-Only Tables

Use this flow when the user already has input rows but no expected labels.

When to Use

• Evaluate or optimize workflows require labeled data, but source table is input-only.
• User wants labels generated and reviewed before full run, with optional model override.

Requirements

• No pre-existing function is required.
• Define output shape via OUTPUT_SCHEMA (JSON object with properties) or FUNCTION_NAME (infers schema from an existing structured function's response_format).

Pseudo-Label Workflow

1. Confirm source table and input columns.
2. Confirm task description for expected-label generation.
3. Define output shape (OUTPUT_SCHEMA or FUNCTION_NAME).
4. Choose destination labeled table (same customer schema).
5. Choose pseudo-label model (default claude-opus-4-6; allow user override).
6. Run a preview with a small row cap.
7. Show preview (INPUT_*, EXPECTED) and allow revise/regenerate.
8. On approval, run full labeling (all rows) and overwrite destination table.
9. Continue to evaluate/optimize with:
   • label_column = EXPECTED
   • input_columns unchanged

Preview Call (sample rows)

Run the same script with --source-table and --max-source-rows to cap the preview:

PYTHONPATH=<SKILL_DIRECTORY>/src uv run --project <SKILL_DIRECTORY> python <SKILL_DIRECTORY>/src/run.py synthetic \
    --database {database} \
    --schema {schema} \
    --connection {connection} \
    --task-description "{task_description}" \
    --output-table {output_table} \
    --input-columns {input_columns...} \
    --model none \
    --num-examples 50 \
    --source-table {source_table} \
    --function-name {function_name or none} \
    --output-schema {output_schema_json or none} \
    --max-source-rows 20

Full Run Call (all rows, overwrite table)

PYTHONPATH=<SKILL_DIRECTORY>/src uv run --project <SKILL_DIRECTORY> python <SKILL_DIRECTORY>/src/run.py synthetic \
    --database {database} \
    --schema {schema} \
    --connection {connection} \
    --task-description "{task_description}" \
    --output-table {output_table} \
    --input-columns {input_columns...} \
    --model {model or none} \
    --num-examples 50 \
    --source-table {source_table} \
    --function-name {function_name or none} \
    --output-schema {output_schema_json or none} \
    --max-source-rows none

Output table shape (same as synthetic mode):

• Input columns (VARCHAR)
• EXPECTED (VARIANT, typed JSON object)

Parameters Reference

Output Table Schema

The SPROC creates a table with the following columns:

Input type handling: All input columns are stored as VARCHAR. When the function has ARRAY-typed parameters and --function-name is provided, the generator automatically produces JSON array values (e.g., ["NAME", "EMAIL"]) instead of plain comma-separated strings. These JSON array strings are correctly detected and parsed by the optimizer and evaluator during downstream workflows.

Accessing output values: Use Snowflake's VARIANT syntax to access individual output keys:

SELECT EXPECTED:verdict, EXPECTED:rationale FROM {output_table};
-- Or cast to string: EXPECTED:verdict::STRING

Return Value

{
    "success": true,
    "output_table": "MY_DB.MY_SCHEMA.SYNTHETIC_DATA",
    "total_generated": 50,
    "input_columns": ["CLAIM", "DOCUMENTS"],
    "expected_keys": ["VERDICT", "RATIONALE"],
    "batch_errors": null
}

Information Checklist

Before calling GENERATE_SYNTHETIC_DATA, ensure you have collected:

Stopping Points

• ✋ Step 1: After inferring/collecting task description — skip when the user named the function up front and inference succeeded
• ✋ Step 4: After model selection
• ✋ Step 6: After showing results
• ✋ Step 7: After presenting next steps
• ✋ Pseudo-Label Step 7: After showing preview for approval

Output

• Snowflake table with synthetic or pseudo-labeled data
• Columns: ID, input columns (VARCHAR), EXPECTED (VARIANT)