Validate Pyspark To Snowpark API

Validate PySpark → Snowpark API (Post-Conversion Pipeline)

Bundled sub-skill under snowpark-api/. This file is loaded via the Read tool by either snowpark-api/SKILL.md (router) or snowpark-api/migrate-pyspark-to-snowpark-api/SKILL.md (after SMA finishes). Not registered as a standalone skill.

Owns the entire post-conversion tail for the Snowpark API path:

1. Initialize git and verify SMA output
2. Generate / re-open the EWI dashboard
3. Run Notebook Migration (optional)
4. Run the EWI Fixer (optional)
5. Run Stage Conversion (optional)
6. Run the DVP Orchestrator (steps 1–13) (optional)
7. Re-open the dashboard
8. Print the Final Summary

This sub-skill is also the direct entry point for <intent> = already_migrated and for individual API-side intents ("fix ewis", "open sma dashboard", "run stage conversion", "resume dvp").

Inputs (passed inline)

Reference Material

• ../references/post-conversion-options.md — step-by-step gate/skip semantics
• ../references/final-summary-template.md — verbatim template for the Final Summary
• ../references/sma-api-reference.md — function reference for sma_api.py
• ../references/configuration-schema.md — config keys this sub-skill reads/writes
• ../references/output-layouts.md — used when <intent>=already_migrated
• ../references/example-workflows.md — concrete walkthroughs

Step V0: Resolve <output> (already_migrated only)

If <intent> = migrate, skip to V1 — <output> is already resolved by the migrate sub-skill.

If <intent> = already_migrated:

1. If <output> was passed in but is not yet at the workload root, apply the resolution rules from ../references/output-layouts.md (v1/v2/v3 in order). Update <output> to the resolved path.
2. If <output> was NOT provided, ask the user (pre-filled with <config.output_folder> as default).
3. Verify:

   test -d "<output>/Output" && test -d "<output>/Reports" && \
       test -f "<output>/Reports/Issues.csv"
   

   If any check fails, ask the user for the correct path. Do not proceed.

Step V1: Initialize Git and Verify SMA Output

cd "<output>"
git rev-parse --is-inside-work-tree 2>/dev/null

If NOT a git repository:

git init
git add .
git commit -m "Initial commit: SMA output before migration process"
git branch -M main

⛔ git branch -M main is MANDATORY — the unmodified SMA output must live on main.

If already a git repository:

• git status --porcelain — if clean, skip to branch creation
• If uncommitted changes, ask the user via ask_user_question:
  • Stash changes: git stash push -m "Pre-migration stash" (recover later with git stash pop)
  • Commit changes: stage + commit pending changes before proceeding
  • Abort: stop and let the user resolve manually

Create / checkout the working branch (idempotent via sma_api.git_ensure_branch):

sma_api.git_ensure_branch("<output>", "sma/migration-process")

All subsequent file modifications happen on sma/migration-process.

Verify output structure (one final check):

test -d "<output>/Output" && test -f "<output>/Reports/Issues.csv"

Step V2: Generate (or Re-Generate) the EWI Dashboard

cd "<output>"

Load the bundled dashboard generator:

SMA_DASHBOARD_SKILL="<snowpark_api_root>/sma-dashboard-generator/SKILL.md"

Read it with the Read tool and follow its instructions in the current <output> working directory. The dashboard generator will:

• Parse Reports/Issues.csv
• Build/refresh sma_storage.sqlite3
• Generate an interactive EWI tracking dashboard at <output>/sma-dashboard/
• Start a local server and open it in the browser

⛔ Do NOT call skill("sma-dashboard-generator") — bundled sub-skill, not registered.

For "open sma dashboard" intent: if <output>/sma-dashboard/ already exists and is recent, you may skip generation and jump straight to V7 (re-open). Otherwise run the generator.

Step V3: Run Notebook Migration (Optional)

Fetch the API-namespaced config slice once and refer to its keys below:

python3 '<spark_migration_root>/scripts/config_manager.py' view-section '<config_path>' snowpark_api

Also load run_notebook_migration from the shared namespace.

Decision table for run_notebook_migration:

Scan-first path (unset): use the detection script from snowflake-notebook-migration:

uv run --project "<spark_migration_root>/snowflake-notebook-migration" \
    python "<spark_migration_root>/snowflake-notebook-migration/scripts/detect_and_parse_notebook.py" \
    --scan "<output>/Output/"

• Empty array → silently skip (no prompt). Go to V4.
• Non-empty → ask_user_question:
  • Yes, run Notebook Migration
  • No, skip for now — persist {"run_notebook_migration": "no"} then go to V4.

Invocation (yes path):

NB_MIGRATION_SKILL="<spark_migration_root>/snowflake-notebook-migration/SKILL.md"

Read with the Read tool and follow inline (foreground only — never run_in_background). Pass this context block:

The following context was configured by the spark-migration orchestrator:
- SMA output directory: <output>
- Notebooks source: <output>/Output/
- Git branch: sma/migration-process (already checked out)
- Conversion type: snowpark-api
- Jupyter conversion: <config.enable_jupyter_conversion>

Scan <output>/Output/ for notebook files (.ipynb, .python, Databricks .py,
.scala, .sql) and convert in-place.
Use sma_api.git_commit() for git operations on the sma/migration-process branch.

Record the number of notebooks converted for the Final Summary.

Step V4: Run the EWI Fixer (Optional)

Decision table for run_ewi_fixer:

Sub-question A (comments): ask_user_question — mark (keep with [FIXED]/[NOT-FIXED] prefix) or remove (delete after fix).

Sub-question B (scope): only_pending / retry_not_resolved / Specific EWI code / all_reset. If "Specific EWI code", also ask for the code (e.g., SPRKPY1002) and store as <ewi_specific_code> (not persisted).

Persist (only the saved keys, not <ewi_specific_code>):

python3 '<spark_migration_root>/scripts/config_manager.py' save '<config_path>' \
    '{"run_ewi_fixer": "yes", "run_ewi_fixer.ewi_comments": "<comments>", "run_ewi_fixer.ewi_scope": "<scope>"}'

Invocation:

EWI_FIXER_SKILL="<snowpark_api_root>/dvp/dvp-ewi-fixer/SKILL.md"

Read with the Read tool and follow inline (foreground only). Context block:

The following options were already configured by the user:
- SMA output directory: <output>
- EWI comment handling: <comments>
- EWIs to process: <scope>
[- Specific EWI code: <ewi_specific_code>]   (only if applicable)

Skip Step 1 questions and proceed directly to Step 2 with these settings.

The fixer will use the SQLite DB built by V2, scan converted files, resolve EWI comments, and update the DB.

Step V5: Run Stage Conversion (Optional)

Decision table for run_stage_conversion:

Invocation:

STAGE_CONV_SKILL="<snowpark_api_root>/stage-conversion/SKILL.md"

Read with the Read tool and follow inline (foreground only). Context block:

The following context was configured by the spark-migration orchestrator:
- SMA output directory: <output>
- Git branch: sma/migration-process (already checked out by ewi-fixer via sma_api.git_ensure_branch)
- Target files directory: <output>/Output/
- Stage prefix: <config.run_stage_conversion.stage_name> (use as the --prefix value)

Skip Step 6 (git check) — repo and branch are already set up via sma_api git functions.
Work directly on the sma/migration-process branch.
Use the provided stage prefix instead of asking the user in Step 3.

Step V6: Run DVP Orchestrator (Optional)

Read run_dvp_orchestrator from the project config (default yes). Do NOT ask the user — the value is already saved.

Invocation:

DVP_ORCH_SKILL="<snowpark_api_root>/dvp/dvp-orchestrator/SKILL.md"

Read with the Read tool and follow inline (foreground only). Context block:

The following context was configured by the spark-migration orchestrator:
- SMA input directory (PySpark source): <input>
- SMA output directory: <output>   (resolved Conversion-*/sma-output/sma-code-process-* path)
- Conversion type: snowpark-api

Skip Step 1 (Detect SMA Paths) — the paths are already known.
Skip the flavor question in Step 4 — conversion_type is snowpark-api, so the
migrated folder is dvp/02-migrated/.
Proceed directly to Step 2 (Validate SMA Structure).
Execute ALL steps through Step 13 (dvp-test-setup-generator). Do NOT stop early.

The orchestrator may load further bundled dvp-* sub-skills from <snowpark_api_root>/dvp/ — none are registered top-level skills.

Step V7: Re-Open the Dashboard

cd "<output>/sma-dashboard" && python3 start_server.py

start_server.py checks if the server is running, finds an available port, starts the server if needed, and opens the user's default browser. Just run it.

Step V8: Print the Final Summary

⛔ Use the EXACT template at ../references/final-summary-template.md.

Before printing, run the pre-display checklist in that reference. Replace ALL placeholders (N, M, <output>, <project_name>, <start_time>, <end_time>, <duration>, <port>) with real values.

Compute <duration> = current time − <start_time>. Use sma_api.git_verify_branches("<output>") to confirm both main and sma/migration-process exist before listing them.

If <intent> = already_migrated, use the already_migrated variant at the bottom of final-summary-template.md (row 1 changes; rows 2–7 are unchanged).

If a step was skipped, show only its parent row with Skipped and omit sub-steps. If a step failed, show Failed with a brief error message and continue the summary.

Stopping Points

Direct-Intent Behavior

When invoked directly via the router for an individual operation:

For these intents, the Final Summary shows Skipped (not requested) for any step that was deliberately bypassed by the intent — distinct from Skipped (configured as disabled).