Databricks Serverless Migration

Serverless Compute Migration

FIRST: Use the parent databricks-core skill for CLI basics, authentication, and profile selection.

Analyze existing Databricks code for serverless compute compatibility and guide migration from classic clusters. The skill follows a 4-step migration lifecycle: Ingest the workload → Analyze for compatibility → Test via A/B comparison → Validate and iterate.

When to Use This Skill

• Migrating notebooks, jobs, or pipelines from classic compute to serverless
• Checking if existing code is serverless-compatible
• Writing new code that targets serverless compute
• Troubleshooting serverless-specific errors after migration
• Choosing between Performance-Optimized and Standard mode

Where to Run This Skill

This skill is published as an Agent Skill (agentskills.io) and runs in any compatible client:

• Claude Code, Cursor, or any agentskills.io client on your laptop — the default. Install via databricks aitools install or follow the per-client docs.
• Inside a Databricks workspace via Genie Code Agent mode — drop the skill into /Workspace/Users/<you>/.assistant/skills/databricks-serverless-migration/ (per-user) or /Workspace/.assistant/skills/databricks-serverless-migration/ (workspace-wide, admin only). See Install in Databricks Genie Code for the three install methods and the important serverless-compute caveat (the Databricks CLI isn't pre-installed on serverless, so some deploy steps need adjustment).

If you finish a migration analysis for a user who's currently running you from a laptop client, mention the Genie Code option once at the end — many users prefer iterating on migrations inside the workspace where the workload lives.

Understanding Migration Blockers

Migration blockers fall into three categories. Focus your effort on category 2 — that's where this skill helps most.

Decision Tree: Is My Workload Ready?

Workload → Check language
├── R code → Category 3: keep on classic
├── Scala notebook cells → Category 2: port to PySpark/SQL, or compile as a JAR job task
├── Compiled Scala JAR (spark_jar_task, build.sbt/pom.xml/build.gradle) → see references/jar-migration.md
│     ├── Scala 2.12 build?             → recompile against 2.13.16
│     ├── Bundles spark-core/spark-sql? → use databricks-connect % Provided
│     └── Bundles libs on the kernel classpath? → mark % Provided (see the classpath table in references/jar-migration.md)
├── Python / SQL → Continue
    ├── Uses RDD APIs? → Category 2: rewrite to DataFrame API (see fixes below)
    ├── Uses DBFS paths? → Category 2: migrate to UC Volumes
    ├── Uses Hive Metastore? → Category 2: migrate to Unity Catalog (or use HMS Federation)
    ├── Uses df.cache/persist? → Category 1: remove and materialize to Delta (native support coming soon)
    ├── Uses streaming?
    │   ├── ProcessingTime trigger → Category 2: use AvailableNow or migrate to SDP
    │   ├── Continuous trigger → Category 2: use SDP continuous mode
    │   ├── No trigger specified → Category 2: add explicit .trigger(availableNow=True)
    │   └── AvailableNow / Once → Ready ✓
    ├── Uses init scripts? → Category 2: use Environments
    ├── Uses VPC peering? → Category 2: use NCCs / Private Link
    ├── Uses unsupported Spark configs? → Category 2: remove (serverless auto-tunes)
    ├── Uses custom JDBC drivers? → Category 2: use Lakehouse Federation or built-in JDBC
    ├── Uses Docker containers? → Category 3: use Environments for libs, or keep on classic
    └── All clear → Ready for serverless ✓

Migration Workflow

Step 1: Ingest — Gather Workload Context

Confirm the migration target is serverless compute. This skill is purpose-built for classic → serverless migrations. The checks, fixes, and workflow all target the serverless compute architecture (Spark Connect, Environments, NCCs). If the user wants to upgrade between classic DBR versions instead, this skill does not apply — classic DBR upgrades have a different compatibility surface and should follow the standard DBR upgrade guide.

Collect the full picture of what needs to migrate to serverless:

• Read the user's notebook/script files
• Identify the classic cluster configuration (instance type, DBR version, Spark configs, init scripts, libraries)
• Note the networking setup (VPC peering, instance profiles, mounts)
• Understand the workload type: batch job, streaming, interactive notebook, pipeline
• Determine the target: the output is always a serverless compute configuration, not a classic cluster with a newer DBR

Step 1.5: Handle Multi-Notebook Workloads

If the workload spans more than one user notebook (exclude _resources/ setup notebooks from the count), process them one at a time rather than all at once. The agent's context window is finite, and trying to hold the full source of an 8-notebook job in active context while doing analysis, fixes, and migration triggers autocompact thrashing (Claude Code) or equivalent context-overflow failures in other clients.

Procedure:

1. Enumerate first. List every user notebook with its path. Do not read the bodies in full yet beyond a few lines for orientation.
   • Check for bundle_config.py (H4). If the source tree contains a bundle_config.py, parse it and follow any upstream-source declarations (git URLs, S3 paths, init_scripts, git_source, or custom upstream-repo declarations). Clone or fetch each referenced repo into a sibling location and include its notebooks in the enumeration. List "external sources" as a first-class artifact category in the migration plan. Without this step, multi-source demos (e.g., dbt-on-databricks references the upstream dbt-databricks-c360 repo) get mis-enumerated and the real task notebooks are missed.
2. For each notebook, in order: a. Read its full source. b. Run the Step 2 Analyze checklist scoped to this notebook only. c. Record a structured summary in your response or to a scratch file, then drop the raw source from active working memory. The summary must include:
   • notebook_path (relative)
   • detected_patterns[] (pattern IDs from this skill's catalog)
   • blockers[] (Category-3 patterns this notebook hits, if any)
   • migration_steps[] (concrete fixes ordered)
   • unmigratable (bool — true if any blocker has no workaround) d. If you have a writable scratch directory (~/.databricks-migration-skill/scratch/<run-id>/findings/), persist the summary as <notebook-basename>.json. This frees the source from context safely. If not, keep the summary compact in conversation history.
3. Synthesize. After all notebooks have a summary, produce the unified migration plan from the summaries alone. Apply fixes notebook-by-notebook, never re-reading the original source unless required to resolve an ambiguity in a summary.
4. Failure Reporting. Trigger the Failure Reporting Protocol exactly once per workload, not once per notebook. The detected_patterns array in the report aggregates across all notebooks; notebook_characteristics uses summed line counts and the union of language/streaming/ML flags.

Threshold guidance: apply this procedure when the user notebook count is ≥ 3, or when individual notebooks exceed ~5KB of source. For 1–2 small notebooks the single-pass workflow in Steps 2–4 is fine.

Anti-pattern: do NOT attempt a "merge all notebooks into one big file" or "read all notebooks, then act in one mega-turn" strategy. Both defeat the purpose by reintroducing the original context-pressure problem.

Step 2: Analyze — Scan for Serverless Readiness

Read notebooks before running them — do not rely on failed job runs to discover issues. A pre-run scan surfaces incompatibilities faster than iterating on error traces, and many serverless failures (hardcoded catalog references, init scripts, missing dependencies) are easy to spot statically but expensive to debug after a failed run.

Before creating or running any test job:

1. Read every notebook and source file referenced by the job
2. Scan for all hardcoded catalog/schema references (e.g., spark.table("main.schema.table"), spark.sql("... FROM main..."), catalog = "main")
3. Check for dependency patterns: init scripts, local wheel files, custom install functions, %pip install lines
4. Locate any requirements.txt or equivalent and resolve the full dependency set
5. Flag OS-level installs (apt install, yum install) for conversion or escalation

Scan the code for patterns that are incompatible with the serverless compute architecture. These checks are serverless-specific — most of these patterns work fine on classic compute regardless of DBR version. For each issue found, report:

• Category: Which of the 3 blocker categories it falls into
• Severity: Blocker (must fix for serverless) / Warning (should fix) / Info (awareness)
• Pattern: What was detected and where
• Fix: Specific remediation targeting serverless compute

Post-rewrite lint: Cell-magic boundary check (A1)

HIGH IMPACT. Before declaring a migrated notebook ready, run this lint pass on every output cell. Caused 3/7 demos to fail in the dbdemos E2E sweep (hls-readmission, fsi-fraud, retail-c360).

Detect: any cell that contains a # MAGIC %<word> line (e.g., # MAGIC %run, # MAGIC %sql, # MAGIC %md, # MAGIC %pip, # MAGIC %fs). Within that cell, every non-blank line must either start with # MAGIC or be a blank line. If a plain-Python comment (or any other Python code) precedes the # MAGIC %... directive in the same cell, the cell is corrupted: Databricks parses it as Python and %run falls back to IPython line magic, producing errors like File './00-global-setup-v2' not found.

Fix: never prepend plain-Python comments above a # MAGIC %... line within the same cell. Two valid options:

1. Preferred: put migration notes in a separate # MAGIC %md cell above (its own # COMMAND ---------- block).
2. Acceptable: drop the migration note entirely and rely on git/file history.

Example before (corrupted; %run fails with File not found):

# COMMAND ----------

# Migration: relative '%run ../../../_resources/00-global-setup-v2' may not
# resolve under serverless job tasks. Replaced with sibling reference.
# MAGIC %run ./00-global-setup-v2

Example after (clean; %run fires as cell magic):

# COMMAND ----------

# MAGIC %md
# MAGIC Migration: relative %run path replaced with sibling reference.

# COMMAND ----------

# MAGIC %run ./00-global-setup-v2

Category A: Unsupported APIs

Category B: Data Access

Category C: Streaming

Category D: Configuration

Category E: Libraries

Category F: Networking

Category G: Sizing & Debugging

Category H: Job-level config (dbt_task, SDP, multi-source, deploy preconditions)

These checks operate on the job/pipeline spec JSON and on deploy preconditions, not on notebook bodies. Apply them alongside the per-notebook checks in Categories A–G.

Required Output: Serverless Environment Specification

The migration output MUST include a Serverless Environment specification alongside migrated code. Generate this by:

1. Scanning all import statements and %pip install lines to detect required packages
2. Extracting init script pip install commands from the job configuration
3. Producing a JSON block suitable for the Jobs API environments field:

{
  "environment_key": "Default",
  "spec": {
    "client": "2",
    "dependencies": ["mlflow==2.12.1", "scikit-learn==1.3.0", "xgboost==2.0.3"]
  }
}

Important: ML runtime libraries (mlflow, scikit-learn, hyperopt, xgboost, tensorflow, torch, etc.) are NOT pre-installed on serverless compute. They MUST be listed explicitly in the environment spec dependencies. ML runtime is NOT available on serverless — always use Serverless Environments with explicit package dependencies instead.

Step 3: Test — Two-Branch Strategy

Use separate branches for testing and production to keep test-only workarounds out of the code that ships. The test branch is a safe sandbox for experimentation; the production branch contains only changes that production actually needs.

Test branch (serverless-test-{job_name}-{timestamp}): Temporary, no PR needed.

1. Create a branch from your current working branch
2. Set up test data: create sampled copies of upstream tables in a test catalog using job lineage (see test data setup below)
3. Parameterize the catalog so the notebook works with both test and production data (see catalog parameterization pattern below)
4. Apply all compatibility fixes discovered in Step 2
5. Create a serverless test job and run it
6. If it fails, get the error output, debug, fix, and retry
7. Document which changes are test workarounds vs. real compatibility fixes

Production branch (serverless-prod-{job_name}): PR required, created from master.

1. Create a new branch from master (NOT from the test branch)
2. Apply ONLY the real compatibility fixes — no test workarounds
3. Apply job config changes (see job config transformation below)
4. Commit and create a PR

Test Data Setup

When the job reads from production tables, do not point the test job at production data. Instead, create sampled copies of upstream tables in a dedicated test catalog and run the test job against those.

The recommended pattern:

1. Resolve the job's upstream tables from its lineage (or from a static scan of the notebook)
2. For each upstream table, run CREATE TABLE IF NOT EXISTS <test_catalog>.<schema>.<table> AS SELECT * FROM <prod_catalog>.<schema>.<table> LIMIT N (typical N: 10–1000 rows)
3. Keep the schema names identical to production — only the catalog changes
4. Make the operation idempotent: skip tables that already exist, so the setup step is safe to re-run
5. Require a running SQL warehouse and CREATE TABLE permission on the test catalog

With schema names preserved, the same notebook code runs in both environments — only the catalog widget value changes.

Decision Tree: Should This Change Go to Production?

Simple test: Would production fail without this change on serverless? If yes → include. If no → test branch only.

A/B Comparison

After both branches are ready, compare outputs:

# Compare outputs between classic and serverless runs
classic_df = spark.read.table("main.output.classic_results")
serverless_df = spark.read.table("main.output.serverless_results")

assert classic_df.count() == serverless_df.count(), "Row count mismatch"
assert classic_df.schema == serverless_df.schema, "Schema mismatch"
diff = classic_df.exceptAll(serverless_df)
assert diff.count() == 0, f"Found {diff.count()} differing rows"

Temporary bridge configs: If the serverless run fails, you may temporarily set supported Spark configs (like spark.sql.shuffle.partitions) to bridge gaps. Mark these as temporary — remove once the workload stabilizes.

Step 4: Validate — Confirm and Monitor

Once the A/B comparison passes:

1. Merge the production branch PR
2. Switch the production job to serverless compute
3. Monitor cost via system tables (system.billing.usage) and budget policies
4. Remove any temporary bridge configurations
5. Set up budget alerts for cost visibility

Migration Deliverables

At the end of a successful migration run, surface these artifacts so the user can verify the work and inspect the results:

If any deliverable is missing, the migration is incomplete — do not mark it as done.

Stopping Conditions

Do not attempt workarounds for these — surface them to the user and stop:

• Permission failures on source tables, the test catalog, or the workspace
• Category 3 blockers (R code, custom Spark data source JARs, features that require classic compute)
• SQL warehouse or test catalog not available
• Repeated failures (typically 5+) with no new information in the error trace — generate a failure report instead (see Failure Reporting Protocol)

Quick Fixes Reference

Replace DBFS paths with UC Volumes

# BEFORE (classic)
df = spark.read.csv("dbfs:/mnt/datalake/sales/data.csv", header=True)
df.write.parquet("dbfs:/mnt/output/results")

# AFTER (serverless)
df = spark.read.csv("/Volumes/main/sales/raw_data/data.csv", header=True)
df.write.parquet("/Volumes/main/analytics/output/results")

# Replace mounts with external volumes (SQL):
# CREATE EXTERNAL VOLUME main.data.raw_files LOCATION 's3://my-bucket/data/';
# Then use: /Volumes/main/data/raw_files/

# Pandas paths too:
# BEFORE: pd.read_csv("/dbfs/mnt/data/file.csv")
# AFTER:  pd.read_csv("/Volumes/main/data/volume/file.csv")

Replace RDD operations with DataFrames

from pyspark.sql import functions as F

# parallelize + map
# BEFORE:
rdd = sc.parallelize([1, 2, 3])
result = rdd.map(lambda x: x * 2).collect()
# AFTER:
df = spark.createDataFrame([(1,), (2,), (3,)], ["value"])
result = df.select((F.col("value") * 2).alias("value")).collect()

# flatMap (word splitting)
# BEFORE:
words = sc.parallelize(["hello world"]).flatMap(lambda l: l.split(" ")).collect()
# AFTER:
df = spark.createDataFrame([("hello world",)], ["line"])
words = df.select(F.explode(F.split("line", " ")).alias("word")).collect()

# groupByKey
# BEFORE:
rdd = sc.parallelize([("a", 1), ("b", 2), ("a", 3)])
grouped = rdd.groupByKey().mapValues(list).collect()
# AFTER:
df = spark.createDataFrame([("a", 1), ("b", 2), ("a", 3)], ["key", "value"])
grouped = df.groupBy("key").agg(F.collect_list("value").alias("values")).collect()

# mapPartitions → applyInPandas
# BEFORE:
def process_partition(iterator):
    yield sum(iterator)
result = sc.parallelize(range(100), 4).mapPartitions(process_partition).collect()
# AFTER:
import pandas as pd
def process_group(pdf: pd.DataFrame) -> pd.DataFrame:
    return pd.DataFrame({"total": [pdf["id"].sum()]})
result = (spark.range(100).repartition(4)
    .groupBy(F.spark_partition_id())
    .applyInPandas(process_group, schema="total long")
    .collect())

# textFile
# BEFORE: rdd = sc.textFile("/mnt/data/file.txt")
# AFTER:  df = spark.read.text("/Volumes/catalog/schema/volume/file.txt")

# wholeTextFiles
# BEFORE: rdd = sc.wholeTextFiles("/mnt/data/dir/")
# AFTER:  df = spark.read.format("binaryFile").load("/Volumes/catalog/schema/volume/dir/")

Fix streaming triggers

# CRITICAL: Omitting .trigger() defaults to ProcessingTime(0) — not supported on serverless

# BEFORE (fails on serverless — no trigger = ProcessingTime default):
query = df.writeStream.format("delta").outputMode("append").start(path)

# BEFORE (fails — explicit ProcessingTime):
query = df.writeStream.trigger(processingTime="10 seconds").start(path)

# AFTER (serverless compatible):
query = (df.writeStream
    .format("delta")
    .outputMode("append")
    .trigger(availableNow=True)
    .option("checkpointLocation", "/Volumes/main/data/checkpoints/stream1")
    .start("/Volumes/main/data/output/stream1"))
query.awaitTermination()

# With OOM prevention (recommended for large sources):
query = (spark.readStream.format("delta")
    .option("maxFilesPerTrigger", 100)          # Delta/Parquet sources
    .option("maxBytesPerTrigger", "10g")         # Limit data per micro-batch
    .load(input_path)
    .writeStream
    .trigger(availableNow=True)
    .option("checkpointLocation", checkpoint_path)
    .start(output_path))

# Kafka: use maxOffsetsPerTrigger
query = (spark.readStream.format("kafka")
    .option("kafka.bootstrap.servers", "broker:9092")
    .option("subscribe", "topic1")
    .option("maxOffsetsPerTrigger", 100000)      # Kafka-specific
    .load()
    .writeStream.trigger(availableNow=True).start(output_path))

# Auto Loader: use cloudFiles.maxFilesPerTrigger (note the prefix)
query = (spark.readStream.format("cloudFiles")
    .option("cloudFiles.format", "json")
    .option("cloudFiles.maxFilesPerTrigger", 1000)  # cloudFiles. prefix
    .load(landing_path)
    .writeStream.trigger(availableNow=True).start(output_path))

Remove caching

# BEFORE (classic):
df = spark.read.parquet(path)
df.cache()
df.count()  # materialize cache
result1 = df.filter("status = 'active'")
result2 = df.groupBy("region").agg(F.sum("revenue"))

# AFTER (serverless — remove .cache(); native support coming soon):
df = spar

…(truncated)