Migrate Pyspark To Snowpark API

Migrate PySpark to Snowpark API (SMA CLI)

Bundled sub-skill under snowpark-api/. This file is loaded via the Read tool by snowpark-api/SKILL.md (see its "Bundled Sub-skill Loading Convention"). Not registered as a standalone skill.

This sub-skill performs only the SMA CLI conversion step. After the SMA output is resolved, control is handed off to validate-pyspark-to-snowpark-api/SKILL.md inline.

Inputs (passed inline by the parent router)

Reference Material

• ../references/sma-cli-options.md — complete CLI flag reference
• ../references/output-layouts.md — v1/v2/v3 output layouts and resolution rules
• ../references/configuration-schema.md — config keys this sub-skill reads and writes

Step M1: Validate SMA CLI Path

Load the global config to read sma_cli_path:

python3 '<spark_migration_root>/scripts/config_manager.py' load-global '<spark_migration_root>'

• If sma_cli_path is set and valid (test -x "<sma_cli_path>"): store as <sma_cli> and proceed to Step M2.
• If sma_cli_path is set but invalid (binary missing/non-executable): inform the user and re-run the detection scan (see parent spark-migration Step 0.2). Once a new path is validated and saved globally, continue.
• If sma_cli_path is NOT set: go back to spark-migration Step 0.2 (the canonical detection routine) — do NOT scan independently. After it persists a valid path, resume here.

Do not proceed until <sma_cli> is set to a verified executable.

Step M2: Collect CLI-Specific Fields

Fetch the Snowpark API slice of the project config:

python3 '<spark_migration_root>/scripts/config_manager.py' \
    view-section '<config_path>' snowpark_api

Three CLI-specific keys drive the optional flags:

If all three keys are already set in the config, skip asking and use the saved values. Otherwise, ask only for the missing keys via ask_user_question, then persist the answers:

python3 '<spark_migration_root>/scripts/config_manager.py' \
    save '<config_path>' \
    '{"enable_jupyter_conversion": "yes", "sql_flavor": "SparkSql", "generate_checkpoints": "yes"}'

Step M3: Run the SMA CLI

Derive optional flags using the table in ../references/sma-cli-options.md, then run the SMA CLI in the background:

"<sma_cli>" -i "<input>" -o "<output>" \
    -e "<email>" -c "<company>" -p "<project>" -y [optional-flags]

Use the Bash tool with:

• run_in_background: true
• description: "Run SMA conversion in background"

Capture the returned <shell_id>.

⛔ Never run SMA in the foreground — large workloads exceed agent timeouts.

Step M4: Monitor Progress

Poll every 5–10 seconds:

bash_output(bash_id: "<shell_id>")

Surface progress to the user using the patterns in ../references/sma-cli-options.md.

Success criteria: Conversion was successful. and exit code 0.

Failure handling:

• Error: lines or non-zero exit code → stop and surface the error verbatim to the user.
• Stuck for >10 minutes with no new output → ask the user whether to wait or abort. Do not silently retry.
• Common quick fixes before re-running:
  • chmod +x "<sma_cli>" if the binary lost execute permissions
  • Verify no other sma process is already running
  • Confirm <input> is a directory of PySpark files (not a single notebook)

Step M5: Resolve SMA Output Layout

After successful completion, resolve <output> to the actual workload root.

Apply the detection rules from ../references/output-layouts.md in order:

1. v1 (timestamped): if <output>/Conversion-* exists, pick the most recent and set <output> to it.
2. v2 (flat): else if <output>/sma-output/ exists, set <output> to it.
3. v3 (dual): else if <output>/Conversion_SnowparkAPI/sma-code-process-* exists, pick the most recent and set <output> to it. (Snowpark API path only ever uses Conversion_SnowparkAPI; never Conversion_SnowparkConnect.)
4. Otherwise: keep <output> unchanged.

After resolution, verify:

test -d "<output>/Output" && test -d "<output>/Reports" && \
    test -f "<output>/Reports/Issues.csv"

If any check fails, stop. The SMA conversion silently failed. Show the user <output>/Logs/ contents (if present) and ask whether to re-run.

⛔ CRITICAL: Do NOT copy Output/, Reports/, or Logs/ upward. All subsequent steps must work inside the resolved <output> folder.

Log: Detected SMA <v1|v2|v3> format. Resolved output path: <output>

Step M6: Hand Off to the Validator

After successful resolution, load the validator and follow it inline:

VALIDATOR_SKILL="<snowpark_api_root>/validate-pyspark-to-snowpark-api/SKILL.md"
test -f "$VALIDATOR_SKILL" || { \
    echo "MISSING: $VALIDATOR_SKILL — reinstall spark-migration"; \
    exit 1; }

Read $VALIDATOR_SKILL with the Read tool and follow its instructions verbatim, passing this context inline:

The following context was configured by migrate-pyspark-to-snowpark-api:
- <intent>           = migrate
- <input>            = <input>
- <output>           = <output>         (resolved SMA workload root)
- <email>            = <email>
- <company>          = <company>
- <project>          = <project>
- <config_path>      = <config_path>
- <spark_migration_root> = <spark_migration_root>
- <snowpark_api_root>    = <snowpark_api_root>
- <start_time>       = <start_time>
- <sma_layout>       = v1 | v2 | v3
- <conversion_done>  = true

Skip any "ask for output path" steps — <output> is already resolved.
Proceed directly to your Step V1 (initialize git + verify SMA output).
Execute V1 through V8 to completion. Do NOT stop early. Do NOT return to me.

⛔ Do not call skill("validate-pyspark-to-snowpark-api"). It is a bundled sub-skill, not a registered top-level skill.

Control does NOT return to this sub-skill after the validator is invoked — the validator owns the rest of the pipeline (dashboard, notebook migration, EWI fixer, stage conversion, DVP orchestrator, final summary).

Stopping Points

Outputs (handed to the validator)

• Resolved <output> (v1 Conversion-* / v2 sma-output/ / v3 sma-code-process-*)
• <output>/Output/ — Snowpark Python converted code
• <output>/Reports/Issues.csv, InputFilesInventory.csv, ArtifactDependencyInventory.csv
• (optional) <output>/Logs/ — SMA logs
• <config_path> updated with any CLI-specific keys the user just answered