# Snowpark API

> Snowpark Python API conversion path via the SMA CLI. Bundled sub-skill of `spark-migration`, invoked by the parent ONLY when the user explicitly asks for an SMA / Snowpark API rewrite (`snowflake.snowpark`, snake_case `with_column` / `group_by` / `session.create_dataframe`). Owns: SMA CLI detection, conversion, EWI dashboard, notebook migration, EWI fixer, stage conversion, DVP orchestration, and the API-side final summary. Use when: user explicitly asked for SMA / SMA CLI / Snowpark API / `snowflake.snowpark` rewrite, OR user wants to operate on an already-converted SMA output (open dashboard, fix EWIs, run stage conversion, resume DVP). Triggers: snowpark api, sma cli, sma conversion, run sma, snowflake.snowpark rewrite, snake_case snowpark, fix ewis, ewi fixer, sma dashboard, stage conversion, resume dvp, dvp orchestrator, already migrated sma.

- Skill: `snowflake-labs/snowpark-api` (Agent Skill, multi-file: 56 files)
- Install (CLI): `npx skillmds@latest add snowflake-labs/snowpark-api`
- Raw SKILL.md: https://api.skillmd.com/api/skills/snowflake-labs/snowpark-api/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Integrations & APIs
- Author: snowflake-labs (https://skillmd.com/u/snowflake-labs)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/snowflake-labs/snowpark-api

---


# Snowpark API (SMA CLI) Migration

Sub-skill for the **Snowpark Python API** conversion path of `spark-migration`. The SMA CLI converts PySpark to `snowflake.snowpark` (snake_case `with_column`, `group_by`, `session.create_dataframe`).

> **Bundled sub-skill of `spark-migration`.** This SKILL.md is loaded
> on-demand by the parent `spark-migration` skill via the Read tool
> (see its "Sub-skill Loading Convention" section) — it is **not**
> registered as a standalone top-level skill in the Cortex Code skill
> registry, by design, to avoid trigger collisions with its parent.
> Do not call `skill("snowpark-api")`; if you reached this file
> outside of a `spark-migration` flow, start at `spark-migration` instead.

## Default-Path Reminder

⛔ **The default `spark-migration` path is Snowpark Connect (SCOS)** via the sibling `snowpark-connect/` sub-skill, which preserves the PySpark API surface (`withColumn`, `groupBy`, `spark.createDataFrame`). This `snowpark-api/` sub-skill is the **opt-in** alternative, selected only when the user explicitly asks for SMA / Snowpark API.

Do **NOT** route here if the user said "convert spark", "migrate pyspark", or anything that does not name SMA / Snowpark API explicitly. Those go to `snowpark-connect/`.

## When to Use

- User explicitly mentions "SMA", "SMA CLI", "Snowpark API", or "`snowflake.snowpark` rewrite"
- Parent `spark-migration` skill has set `<config.conversion_type> = "snowpark-api"` (or any legacy alias normalized to it: `snowpark_api`)
- User wants to operate on an existing SMA output (already-migrated flow) — open the EWI dashboard, run the EWI fixer, perform stage conversion, or resume DVP

## Inputs (from parent orchestrator)

The parent passes the following inline as the next turn's context when delegating to this sub-skill:

| Variable | Source | Description |
|---|---|---|
| `<input>` | `<config.input_folder>` | PySpark source directory or file |
| `<output>` | `<config.output_folder>` | Target / existing SMA output directory |
| `<email>` | `<config.email>` | Customer email (passed to SMA CLI) |
| `<company>` | `<config.company>` | Customer company (passed to SMA CLI) |
| `<project>` | `<config.project_name>` | Project name (passed to SMA CLI) |
| `<config_path>` | `<spark_migration_root>/configurations/<project>.json` | Project config for persisting SMA-only choices |
| `<global_config>` | Loaded from `<spark_migration_root>/config.json` | Provides `sma_cli_path` if already detected |
| `<start_time>` | Recorded at parent Step 1 | Used for the Final Summary duration |
| `<intent>` | Parent decides | `migrate` (fresh SMA conversion) or `already_migrated` (operate on existing output) |

Sub-skill behavior also depends on `<config.*>` keys (`enable_jupyter_conversion`, `sql_flavor`, `generate_checkpoints`, `run_ewi_fixer`, `run_ewi_fixer.ewi_comments`, `run_ewi_fixer.ewi_scope`, `run_stage_conversion`, `run_stage_conversion.stage_name`, `run_dvp_orchestrator`). Use `config_manager.py view-section snowpark_api` to fetch just this slice.

## Path Resolution

`<spark_migration_root>` resolves to the grandparent of every `snowpark-api/<child>/SKILL.md` — the `spark-migration/` directory that contains the shared `scripts/config_manager.py` and `snowflake-notebook-migration/` skill.

`<snowpark_api_root>` resolves to **this** sub-skill's directory — `spark-migration/snowpark-api/`. The child skills (`migrate-pyspark-to-snowpark-api/`, `validate-pyspark-to-snowpark-api/`, `sma-dashboard-generator/`, `stage-conversion/`, `dvp/`) live underneath it.

```
spark-migration/                                  ← <spark_migration_root>
├── SKILL.md                                      (parent, thin router)
├── scripts/config_manager.py                     (shared)
├── snowflake-notebook-migration/                 (shared)
└── snowpark-api/                                 ← <snowpark_api_root>
    ├── SKILL.md                                  (this file)
    ├── scripts/sma_api.py                        (SMA SQLite/git/EWI helpers)
    ├── references/                               (extracted long-form docs)
    ├── migrate-pyspark-to-snowpark-api/SKILL.md
    ├── validate-pyspark-to-snowpark-api/SKILL.md
    ├── sma-dashboard-generator/SKILL.md
    ├── stage-conversion/SKILL.md
    └── dvp/dvp-*/SKILL.md  (orchestrator + 9 children)
```

## Bundled Sub-skill Loading Convention

This sub-skill loads its children by **reading their `SKILL.md` files** with the Read tool, NOT by registry lookup. Children load each other the same way.

| Intent / role | Bundled path (Read with Read tool, follow inline) |
|---|---|
| Fresh SMA conversion | `<snowpark_api_root>/migrate-pyspark-to-snowpark-api/SKILL.md` |
| Post-conversion tail / already-migrated entry | `<snowpark_api_root>/validate-pyspark-to-snowpark-api/SKILL.md` |
| EWI dashboard generator | `<snowpark_api_root>/sma-dashboard-generator/SKILL.md` |
| Stage path replacement | `<snowpark_api_root>/stage-conversion/SKILL.md` |
| DVP pipeline | `<snowpark_api_root>/dvp/dvp-orchestrator/SKILL.md` |
| Notebook migration (shared, at parent root) | `<spark_migration_root>/snowflake-notebook-migration/SKILL.md` |

If any file is missing, **STOP** and report:
> The bundled `<name>` sub-skill is missing at `<expected_path>`.
> Reinstall the `spark-migration` skill.

## Intent Detection

```
Start (parent has delegated to this sub-skill)
  ↓
Detect Language
  ├─→ Python (.py, PySpark, Databricks, pyspark)
  │     ├─→ <intent>=migrate          → load migrate-pyspark-to-snowpark-api/SKILL.md
  │     └─→ <intent>=already_migrated → load validate-pyspark-to-snowpark-api/SKILL.md
  │
  ├─→ Scala (.scala, Spark Scala, build.sbt)
  │     → Not yet supported via this sub-skill.
  │       Ask the user whether they want to switch to `snowpark-connect`,
  │       which does support Scala.
  │
  └─→ Ambiguous → Ask the user which language the workload uses
```

### Step 1: Detect Language

Determine source language from:
- **Explicit mention**: "PySpark", "Python Spark"
- **File extensions**: `.py` → Python
- **Import patterns**: `from pyspark` / `import pyspark` → Python
- **Build files**: `requirements.txt` / `pyproject.toml` → Python

If the language cannot be determined, ask the user. If the user says Scala, inform them that Snowpark API conversion for Scala is not yet supported via this sub-skill and ask whether they want to switch to `snowpark-connect` (SCOS) instead.

### Step 2: Route by Intent

| `<intent>` | Bundled SKILL.md to load |
|---|---|
| `migrate` | `<snowpark_api_root>/migrate-pyspark-to-snowpark-api/SKILL.md` |
| `already_migrated` | `<snowpark_api_root>/validate-pyspark-to-snowpark-api/SKILL.md` |

Read the chosen file with the Read tool and follow its instructions inline. Pass the full inputs block from the parent's context unchanged.

The migrate sub-skill owns SMA CLI detection + run + output resolution, then **delegates to the validator** to run the post-conversion tail (dashboard → notebook → EWI → stage → DVP → final summary). The validator is also the direct entry point for `already_migrated` flows and for individual API-side intents ("open dashboard", "fix ewis", "run stage conversion", "resume DVP").

Control does NOT return to the parent until the entire pipeline completes (or the user explicitly aborts).

## Config

This sub-skill reads and writes only the `snowpark_api` namespace via:

```bash
python3 '<spark_migration_root>/scripts/config_manager.py' view-section '<config_path>' snowpark_api
```

That returns the SMA-only project keys (`sql_flavor`, `enable_jupyter_conversion`, `generate_checkpoints`, `run_ewi_fixer`, `run_ewi_fixer.ewi_comments`, `run_ewi_fixer.ewi_scope`, `run_stage_conversion`, `run_stage_conversion.stage_name`, `run_dvp_orchestrator`).

The global `sma_cli_path` is read from `<spark_migration_root>/config.json` via `config_manager.py load-global`.

Schema reference: [`references/configuration-schema.md`](references/configuration-schema.md).

## Stopping Points

None at this routing level — stopping points are defined inside `migrate-pyspark-to-snowpark-api/SKILL.md` and `validate-pyspark-to-snowpark-api/SKILL.md`.

## Output

The migrate sub-skill produces an SMA `Conversion-*` (v1) or `sma-output/` (v2) or `Conversion_SnowparkAPI/sma-code-process-*` (v3) folder containing:

- `Output/` — Snowpark Python (snake_case) converted code
- `Reports/Issues.csv`, `Reports/InputFilesInventory.csv`, `Reports/ArtifactDependencyInventory.csv`
- `sma-dashboard/` — interactive EWI tracking dashboard (after validator runs the dashboard generator)
- `dvp/` — validation pipeline workspace (after validator runs the DVP orchestrator)

Plus per-step optional artifacts: EWI fixes, stage-rewritten paths, generated test suites. See `references/output-layouts.md` for the full layout and v1/v2/v3 path resolution rules.

The final summary template lives at [`references/final-summary-template.md`](references/final-summary-template.md) and is rendered by the validator at the end of the pipeline.