# Assess Pyspark Workload > Standalone assessment of PySpark and Databricks workloads for Snowflake SCOS compatibility. Produces a MigrationReadinessReport.html and AssessmentIR.json without committing to migration. Use when: analyzing Spark workload complexity, generating a migration readiness report, understanding migration effort before deciding, evaluating PySpark compatibility with SCOS, assessing a Databricks workload for Snowflake, or producing a pre-migration feasibility report. - Skill: `snowflake-labs/assess-pyspark-workload` (Agent Skill) - Install (CLI): `npx skillmds@latest add snowflake-labs/assess-pyspark-workload` - Raw SKILL.md: https://api.skillmd.com/api/skills/snowflake-labs/assess-pyspark-workload/raw - Safety review: WARNING - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Web & Frontend - Author: snowflake-labs (https://skillmd.com/u/snowflake-labs) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/snowflake-labs/assess-pyspark-workload --- # Assess PySpark Workload — Coordinator Scan a PySpark or Databricks workload, run SCOS compatibility analysis, and produce a stakeholder-facing readiness report — without starting a migration. The report can be used as a standalone deliverable or as the launchpad to continue into the full SCOS migration. This coordinator runs phases 0 through 1.2 of the SCOS migration pipeline (workspace setup, deterministic pre-processing, compatibility analysis, and report rendering) using the same phase playbooks and scripts as the full migration coordinator. There is no separate assessment implementation — the two coordinators share every phase file under `phases/` and every agent under `agents/`. ## When to Load [snowpark-connect] Intent Detection: After user indicates assessment or pre-migration analysis intent (assess, readiness report, analyze workload, understand migration effort, evaluate compatibility, check before migrating, how hard is migration, migration feasibility). ## Arguments - `$ARGUMENTS` — Path to the PySpark file or directory to assess ## Placeholders | Placeholder | Set by | Description | |-------------|--------|-------------| | `` | step 1 | Absolute path to `snowpark-connect/` (contains `pyproject.toml` and `scripts/`) | | `` | step 1 | `/migrate-pyspark-to-snowpark-connect` — where phase playbooks and agents live | | `` | Phase 0 | Timestamped workspace root, e.g. `/Conversion-SCOS-` | | `coordinator_mode` | Phase 0 | `false` for single-file; `true` for multi-file | When a phase playbook references `../agents/.md`, resolve it as `/agents/.md`. When it references `../SKILL.md`, resolve it as `/SKILL.md` (the migration coordinator). ## Prerequisites ### Skill Directory `` is the `snowpark-connect/` directory that contains `pyproject.toml` and `scripts/`. It is the parent of both this file's directory and `migrate-pyspark-to-snowpark-connect/`. Resolve it as the grandparent of this `SKILL.md`. All script invocations use `uv run --project `. `` = `/migrate-pyspark-to-snowpark-connect` ### uv Package Manager Install `uv` if not already available. Show both OS variants: ```bash # macOS / Linux uv --version || curl -LsSf https://astral.sh/uv/install.sh | sh ``` ```powershell # Windows (PowerShell) uv --version; if ($LASTEXITCODE -ne 0) { irm https://astral.sh/uv/install.ps1 | iex } ``` ## State Model State lives in `/migration_state.json` — the same file and schema used by the migration coordinator. The phase keys written here (`0_5_preprocess`, `1_analysis`, `1a_assessment_report`, etc.) are the same keys the migration coordinator reads when it resumes from an existing assessment. Do not invent alternative keys. ## Workflow ### Step 1: Collect Info Ask for the two fields upfront in one message: ``` To run the assessment I need: 1. Source path: (PySpark file or directory to assess) 2. Project name: (used to label the report — defaults to the last path component) ``` Output path defaults to `_scos`. Email and company are not needed for the assessment report; leave them blank in `migration_state.json :: metadata` for now. ### Step 2: Phases 0 through 1.2 Run each phase in order by reading its playbook and following the instructions exactly. > **Gate contract:** the same universal gate contract from the migration coordinator applies. > Exit `0` = advance. Exit `2` = gap-scoped retry (3× for Phase 1.2, 2× for Phase 1). > Exit `3` = stop and escalate. #### Phase 0 — Workspace setup Read `/phases/phase-0-setup.md` and follow it. Pass: `$ARGUMENTS` = source path, `$OUTPUT` = ``, `$PROJECT` = project name. Email and company may be omitted from `migration_state.json :: metadata` at this stage. #### Phase 0.5 + 0.6 — Deterministic pre-processing Read `/phases/phase-0.5-preprocess.md` and follow it. (This file covers both Phase 0.5 recipe pre-processing and the Phase 0.6 SQL rewrite.) #### Phase 1 + 1.1 — Compatibility analysis and adjudication Read `/phases/phase-1-analysis.md` and follow it. (This file covers Phase 1 analysis and the conditional Phase 1.1 adjudication pass.) #### Phase 1.2 — Assessment report Read `/phases/phase-1.2-assessment.md` and follow it. (This file covers Phase 1.2 report rendering and the optional Phase 1.3 data-edge enrichment.) **Stop here.** Do not read or load Phase 2 playbooks. ### Step 3: Present the Report After Phase 1.2 gate passes, present the result in chat: ``` ✅ Assessment complete. Report: /Reports/MigrationReadinessReport.html IR: /Reports/AssessmentIR.json Open the HTML report in a browser for the full five-tab view: Overview · Detailed Compatibility · Migration Plan · API Compatibility · Discovery ``` Describe the migration effort using the code-churn **categories** — Ready / Light Refactor / Active Refactor — and the per-bucket file counts. **Never quote a numeric readiness score or percentage** — the assessment is deliberately category-based. ### Step 4: Offer Migration Upgrade Ask in one message: ``` Would you like to continue with the full SCOS migration from this assessment? The workspace at is already set up — migration picks up at Phase 2 (code fixes, import updates, final reports). No re-scanning or re-processing. Continue? (Y/n) ``` **Yes** — Read `/SKILL.md` and follow it, passing `$ARGUMENTS`, `$OUTPUT`, and `$PROJECT`. The migration coordinator's startup probe will discover the completed assessment at `` from disk and resume from Phase 2 automatically — no explicit handoff parameter needed. Email and company were not collected during assessment; the migration coordinator will prompt for them in Phase 4. **No** — Exit cleanly: ``` The assessment is saved at . To continue the migration later, re-invoke the SCOS migration skill and point it at — it will detect the existing assessment and resume from Phase 2. To re-run LLM data-edge enrichment: uv run --project \ python /scripts/assessment/render_assessment.py \ --llm-resolved-edges \ --dump-ir /Reports/AssessmentIR.json \ --output-html /Reports/MigrationReadinessReport.html (--dump-ir is required; without it the flag loads nothing and the report is static-only) ``` ## Stopping Points - After Step 1: confirm all settings with the user before Phase 0 executes (unconditional) - After Step 1: if source path contains > 100 Python files, additionally confirm scale before proceeding - After Phase 1 gate FAIL × 2: escalate to user before attempting a third retry - After Phase 1.2 gate FAIL × 3: STOP and escalate — do not present a result without a valid report ## Output ``` / Conversion-SCOS-/ ← Output/ ← source copy (pre-recipe state in git tag phase-0-source) Reports/ MigrationReadinessReport.html ← stakeholder-facing five-tab readiness report AssessmentIR.json ← structured IR (stable contract for downstream tooling) migration_state.json ← phase gate tracking (reusable if migration continues) analysis.json ← compatibility findings ```