Deploying Databricks jobs and pipelines on serverless
FIRST: Use the parent databricks-core skill for CLI basics, authentication and profile selection.
This skill decides compute. Where another skill or example (for example databricks-jobs) shows
cluster configuration, follow this skill instead.
Serverless is the default compute for every job, pipeline and bundle you create. If the user
explicitly asks for a cluster, do what they ask and mention the serverless option in one line.
When editing an existing job, keep its compute unless the user asks to change it; tasks you add
follow this skill.
1. Serverless or classic
Use classic compute only for:
- R code, or Scala in notebook cells (
%scala): serverless notebooks run Python and SQL only;
- custom Docker images;
- OS packages (
apt-get) or native libraries with no pip equivalent;
- custom Spark data source JARs;
- a stated hard requirement for an instance type or Databricks Runtime version;
- an explicit user request;
- existing code over the budget in section 5;
- a workspace without serverless: if creating or running the job fails because serverless is not
available there, put the whole job on classic and say so.
Then give only the blocked task a cluster (one small job_clusters entry with the latest LTS
runtime and autoscale: {min_workers: 1, max_workers: 4}), keep every other task serverless, and
state the blocker in one line. For a blocked pipeline, give the pipeline a clusters block instead
of serverless: true.
Not reasons for classic: Scala or Java in a JAR task (JAR tasks run on serverless), GPUs (serverless
GPU), large data, long runtimes, Kafka, cost (serverless STANDARD is on par with or cheaper than
on-demand classic in most cases), or habit.
2. Put every task on serverless
3. Choose the performance mode
Set the job-level performance_target:
STANDARD when nothing is time-critical and cost matters more than speed: scheduled batch and
ETL, nightly or weekly runs, or any job where the user states no latency need. Runs start within
minutes instead of seconds and cost less. This is the default choice.
PERFORMANCE_OPTIMIZED only when the user names an SLA or a deadline that is tight relative
to the runtime, a person waits on the result, or the job runs every 30 minutes or more often.
performance_target applies to the whole job, notebook tasks included (only interactive notebooks
always run performance-optimized). Say which mode you chose and why in one line, e.g. "STANDARD:
nightly batch, no deadline."
Serverless GPU tasks ignore performance_target.
Pipelines have no performance_target of their own. To schedule a pipeline, trigger it from a job
with a pipeline_task and set the mode on that job; the job's mode applies to the pipeline update.
A continuous pipeline runs in STANDARD only when a continuous job runs it.
4. Code you write runs on serverless from the start
Serverless reads Unity Catalog tables, /Volumes/... paths, cloud paths behind a Unity Catalog
external location, and the sample data under dbfs:/databricks-datasets/. It cannot read DBFS
mounts (dbfs:/mnt/...) or other DBFS root paths; only those count as a DBFS blocker.
DataFrame and SQL APIs only (no RDDs, sc.*, spark.sparkContext); Unity Catalog three-part table
names; /Volumes/... paths instead of DBFS mounts (dbfs:/mnt/...); availableNow streaming
triggers; no .cache() or .persist(); no spark.conf.set for executor, driver or memory
settings; pinned dependencies in the environment, never init scripts.
5. Existing code: a quick check, not a migration
The user is waiting for the deployment. Spend one short pass, then deploy.
Scan only the files the job runs (a text search such as grep or rg; do not read the whole
repo) for: sc., sparkContext,
.rdd, parallelize, mapPartitions, reduceByKey, groupByKey, dbfs:/, /dbfs/,
dbutils.fs.mount, .cache(, .persist(, spark.conf.set, processingTime, continuous=,
writeStream, hive_metastore., GLOBAL TEMP, hivevar, init_scripts, apt-get, docker,
and other languages: %scala or %r as the whole magic (# MAGIC %r, not %run), .r files
and R notebooks, SparkR and sparklyr. R and Scala cells go to classic (section 1).
Minor blockers: fix them and deploy on serverless if every fix is mechanical and local, keeps
the logic, needs no new infrastructure, and there are about 5 small edits or fewer in total:
- delete
.cache(), .persist(), .unpersist(), setLogLevel, executor/driver/AQE spark.conf.set lines
- move
pip installs or requirements.txt into the environment, pinned
- add
.trigger(availableNow=True) to a stream that sets no trigger, in a job that runs on a
schedule anyway (a stream with a processingTime or continuous trigger is a step 4 blocker)
- replace a
dbfs:/mnt/... path with a /Volumes/... path only when that volume's storage
location is the mount's source, so the job reads the same data; a volume with a similar name is
not enough. Otherwise the mount is a step 4 blocker.
CREATE GLOBAL TEMP VIEW becomes CREATE TEMP VIEW, and its global_temp.<view> reads become
<view>, when the same file creates and reads the view (serverless has no global temp views); a
global temp view that another task or notebook reads is a step 4 blocker
- add the catalog to a two-part table name only when you know which Unity Catalog catalog holds the
table (for example the job's default catalog); a table in
hive_metastore is a step 4 blocker
List every edit you made in your reply.
RDD or SparkContext code: translate it to DataFrames and deploy on serverless only if all of
these hold:
- it is one file of about 200 lines or fewer;
- the change is a pure API translation: the same logic, expressed with DataFrame or SQL;
- every input path, table, output table and parameter stays exactly as it was. Never swap in
another dataset, even a sample one;
- every input is already readable on serverless (section 4: a Unity Catalog table, an existing
/Volumes/... path, a cloud path behind an external location, or dbfs:/databricks-datasets/),
so nothing new has to be set up.
Save the original next to it as <name>.classic.py and point the job at the translated file.
Tell the user that the translation has not been run yet and suggest one test run that compares
the output with the classic version.
Anything bigger: do not rewrite. Put that task on classic now (section 1), keep the other
tasks serverless, and end with: "Kept on classic because . To move it to
serverless, use the databricks-serverless-migration skill." Bigger means: RDD code that fails any
condition in step 3 (for example it reads a DBFS mount), mounts with no volume on the same
storage, Hive Metastore to Unity Catalog, recompiling a JAR, networking changes, changing what a
stream does (an always-on processingTime or continuous stream), or more than about 5 edits
besides a step 3 translation.
Do not stop to ask whether to migrate. Deploy, then report what you did.
Common Issues
| Issue |
Solution |
Libraries field is not supported for serverless task |
Move libraries into environments[].spec.dependencies (JARs: java_dependencies) and set environment_key on the task |
| Serverless job starts too slowly |
performance_target: PERFORMANCE_OPTIMIZED, only if the user needs fast starts (section 3) |
Only remote Spark sessions ... are supported or another RDD error |
RDD/SparkContext code: section 5, step 3 or 4 |
[PATH_NOT_FOUND] or no access on dbfs:/mnt/... |
DBFS mounts do not work on serverless: use a volume on the same storage (section 5, step 2), otherwise step 4 |
| Serverless is not enabled or not available in the workspace |
Put the job on classic and tell the user (section 1) |
A sibling skill's example uses job_clusters for a new job |
Leave the cluster block out; this skill decides compute. Existing jobs keep their compute |
Related Skills
- databricks-core - CLI basics, authentication, profiles
- databricks-jobs - job structure, task types, triggers, notifications
- databricks-pipelines - Lakeflow Spark Declarative Pipelines
- databricks-serverless-migration - migrating existing classic workloads to serverless
Documentation
1---2name: databricks-serverless3description: "Serverless compute for Databricks jobs, Lakeflow pipelines and Declarative Automation Bundles (DABs), with STANDARD or PERFORMANCE_OPTIMIZED and classic only where serverless cannot run the workload. Use when creating, deploying, scheduling or editing a job or pipeline, or deciding its compute. Invoke BEFORE writing a job spec. For migrating existing classic workloads, use databricks-serverless-migration."4compatibility: Requires databricks CLI (>= v0.292.0)5metadata:6 source: https://skillmd.com/skills/databricks/databricks-serverless7 version: "0.3.2"8parent: databricks-core9---1011# Deploying Databricks jobs and pipelines on serverless1213**FIRST**: Use the parent `databricks-core` skill for CLI basics, authentication and profile selection.1415This skill decides compute. Where another skill or example (for example `databricks-jobs`) shows16cluster configuration, follow this skill instead.1718Serverless is the default compute for every job, pipeline and bundle you create. If the user19explicitly asks for a cluster, do what they ask and mention the serverless option in one line.20When editing an existing job, keep its compute unless the user asks to change it; tasks you add21follow this skill.2223## 1. Serverless or classic2425Use classic compute only for:2627- R code, or Scala in notebook cells (`%scala`): serverless notebooks run Python and SQL only;28- custom Docker images;29- OS packages (`apt-get`) or native libraries with no pip equivalent;30- custom Spark data source JARs;31- a stated hard requirement for an instance type or Databricks Runtime version;32- an explicit user request;33- existing code over the budget in section 5;34- a workspace without serverless: if creating or running the job fails because serverless is not35 available there, put the whole job on classic and say so.3637Then give only the blocked task a cluster (one small `job_clusters` entry with the latest LTS38runtime and `autoscale: {min_workers: 1, max_workers: 4}`), keep every other task serverless, and39state the blocker in one line. For a blocked pipeline, give the pipeline a `clusters` block instead40of `serverless: true`.4142Not reasons for classic: Scala or Java in a JAR task (JAR tasks run on serverless), GPUs (serverless43GPU), large data, long runtimes, Kafka, cost (serverless STANDARD is on par with or cheaper than44on-demand classic in most cases), or habit.4546## 2. Put every task on serverless4748- **New jobs:** leave out `new_cluster`, `job_clusters`, `job_cluster_key`, `existing_cluster_id`,49 `instance_pool_id`, `node_type_id`, `num_workers`, `autoscale`, `spark_version`. A task without a50 cluster runs on serverless. In an existing job, keep the cluster settings it has.51- **Notebook tasks** need nothing else. **Python script, wheel and JAR tasks** take their libraries52 from a job-level environment. Use the latest environment version the workspace offers, as a53 quoted string:54 ```yaml55 environments:56 - environment_key: default57 spec:58 environment_version: "4" # example; use the latest59 dependencies: ["requests==2.32.3"] # pin versions; JARs go in java_dependencies60 tasks:61 - task_key: main62 spark_python_task: {python_file: /Workspace/.../main.py}63 environment_key: default64 ```65- **SQL tasks:** a serverless SQL warehouse. **Pipelines:** `serverless: true`, no `clusters` block.66- **Streaming:** `.trigger(availableNow=True)`, the trigger serverless supports (no `processingTime`67 or `continuous` triggers). For always-on processing, run that stream in a continuous job68 (`continuous: {pause_status: UNPAUSED}`), which starts the next run as soon as one finishes, or use69 a continuous pipeline.70- **Tag every job you create** for resource tracking, as other Databricks skills do: job-level71 `tags: {"aidevkit_project": "ai-dev-kit"}`. Keep tags the user already has.7273## 3. Choose the performance mode7475Set the job-level `performance_target`:7677- **`STANDARD`** when nothing is time-critical and cost matters more than speed: scheduled batch and78 ETL, nightly or weekly runs, or any job where the user states no latency need. Runs start within79 minutes instead of seconds and cost less. This is the default choice.80- **`PERFORMANCE_OPTIMIZED`** only when the user names an SLA or a deadline that is tight relative81 to the runtime, a person waits on the result, or the job runs every 30 minutes or more often.8283`performance_target` applies to the whole job, notebook tasks included (only interactive notebooks84always run performance-optimized). Say which mode you chose and why in one line, e.g. "STANDARD:85nightly batch, no deadline."86Serverless GPU tasks ignore `performance_target`.8788Pipelines have no `performance_target` of their own. To schedule a pipeline, trigger it from a job89with a `pipeline_task` and set the mode on that job; the job's mode applies to the pipeline update.90A continuous pipeline runs in STANDARD only when a continuous job runs it.9192## 4. Code you write runs on serverless from the start9394Serverless reads Unity Catalog tables, `/Volumes/...` paths, cloud paths behind a Unity Catalog95external location, and the sample data under `dbfs:/databricks-datasets/`. It cannot read DBFS96mounts (`dbfs:/mnt/...`) or other DBFS root paths; only those count as a DBFS blocker.9798DataFrame and SQL APIs only (no RDDs, `sc.*`, `spark.sparkContext`); Unity Catalog three-part table99names; `/Volumes/...` paths instead of DBFS mounts (`dbfs:/mnt/...`); `availableNow` streaming100triggers; no `.cache()` or `.persist()`; no `spark.conf.set` for executor, driver or memory101settings; pinned dependencies in the environment, never init scripts.102103## 5. Existing code: a quick check, not a migration104105The user is waiting for the deployment. Spend one short pass, then deploy.1061071. **Scan only the files the job runs** (a text search such as `grep` or `rg`; do not read the whole108 repo) for: `sc.`, `sparkContext`,109 `.rdd`, `parallelize`, `mapPartitions`, `reduceByKey`, `groupByKey`, `dbfs:/`, `/dbfs/`,110 `dbutils.fs.mount`, `.cache(`, `.persist(`, `spark.conf.set`, `processingTime`, `continuous=`,111 `writeStream`, `hive_metastore.`, `GLOBAL TEMP`, `hivevar`, `init_scripts`, `apt-get`, `docker`,112 and other languages: `%scala` or `%r` as the whole magic (`# MAGIC %r`, not `%run`), `.r` files113 and R notebooks, `SparkR` and `sparklyr`. R and Scala cells go to classic (section 1).1142. **Minor blockers: fix them and deploy on serverless** if every fix is mechanical and local, keeps115 the logic, needs no new infrastructure, and there are about 5 small edits or fewer in total:116 - delete `.cache()`, `.persist()`, `.unpersist()`, `setLogLevel`, executor/driver/AQE `spark.conf.set` lines117 - move `pip install`s or `requirements.txt` into the environment, pinned118 - add `.trigger(availableNow=True)` to a stream that sets no trigger, in a job that runs on a119 schedule anyway (a stream with a `processingTime` or `continuous` trigger is a step 4 blocker)120 - replace a `dbfs:/mnt/...` path with a `/Volumes/...` path only when that volume's storage121 location is the mount's source, so the job reads the same data; a volume with a similar name is122 not enough. Otherwise the mount is a step 4 blocker.123 - `CREATE GLOBAL TEMP VIEW` becomes `CREATE TEMP VIEW`, and its `global_temp.<view>` reads become124 `<view>`, when the same file creates and reads the view (serverless has no global temp views); a125 global temp view that another task or notebook reads is a step 4 blocker126 - add the catalog to a two-part table name only when you know which Unity Catalog catalog holds the127 table (for example the job's default catalog); a table in `hive_metastore` is a step 4 blocker128129 List every edit you made in your reply.1303. **RDD or SparkContext code: translate it to DataFrames and deploy on serverless** only if all of131 these hold:132 - it is one file of about 200 lines or fewer;133 - the change is a pure API translation: the same logic, expressed with DataFrame or SQL;134 - every input path, table, output table and parameter stays exactly as it was. Never swap in135 another dataset, even a sample one;136 - every input is already readable on serverless (section 4: a Unity Catalog table, an existing137 `/Volumes/...` path, a cloud path behind an external location, or `dbfs:/databricks-datasets/`),138 so nothing new has to be set up.139140 Save the original next to it as `<name>.classic.py` and point the job at the translated file.141 Tell the user that the translation has not been run yet and suggest one test run that compares142 the output with the classic version.1434. **Anything bigger: do not rewrite.** Put that task on classic now (section 1), keep the other144 tasks serverless, and end with: "Kept <task> on classic because <blocker>. To move it to145 serverless, use the databricks-serverless-migration skill." Bigger means: RDD code that fails any146 condition in step 3 (for example it reads a DBFS mount), mounts with no volume on the same147 storage, Hive Metastore to Unity Catalog, recompiling a JAR, networking changes, changing what a148 stream does (an always-on `processingTime` or `continuous` stream), or more than about 5 edits149 besides a step 3 translation.150151Do not stop to ask whether to migrate. Deploy, then report what you did.152153## Common Issues154155| Issue | Solution |156|-------|----------|157| `Libraries field is not supported for serverless task` | Move libraries into `environments[].spec.dependencies` (JARs: `java_dependencies`) and set `environment_key` on the task |158| Serverless job starts too slowly | `performance_target: PERFORMANCE_OPTIMIZED`, only if the user needs fast starts (section 3) |159| `Only remote Spark sessions ... are supported` or another RDD error | RDD/SparkContext code: section 5, step 3 or 4 |160| `[PATH_NOT_FOUND]` or no access on `dbfs:/mnt/...` | DBFS mounts do not work on serverless: use a volume on the same storage (section 5, step 2), otherwise step 4 |161| Serverless is not enabled or not available in the workspace | Put the job on classic and tell the user (section 1) |162| A sibling skill's example uses `job_clusters` for a new job | Leave the cluster block out; this skill decides compute. Existing jobs keep their compute |163164## Related Skills165166- **databricks-core** - CLI basics, authentication, profiles167- **databricks-jobs** - job structure, task types, triggers, notifications168- **databricks-pipelines** - Lakeflow Spark Declarative Pipelines169- **databricks-serverless-migration** - migrating existing classic workloads to serverless170171## Documentation172173- [Serverless compute for jobs](https://docs.databricks.com/jobs/run-serverless-jobs)174- [Serverless compute limitations](https://docs.databricks.com/compute/serverless/limitations)175- [Serverless dependencies (environments)](https://docs.databricks.com/compute/serverless/dependencies)176- [Serverless pipelines](https://docs.databricks.com/ldp/serverless)