databrickslabs Databricks Execution & Compute
Run code on Databricks. Three execution modes—choose based on workload.
Execution Mode Decision Matrix
| Aspect | Databricks Connect ⭐ | Serverless Job | Interactive Cluster |
|---|---|---|---|
| Use for | Spark code (ETL, data gen) | Heavy processing (ML) | State across tool calls, Scala/R |
| Startup | Instant | ~25-50s cold start | ~5min if stopped |
| State | Within Python process | None | Via context_id |
| Languages | Python (PySpark) | Python, SQL | Python, Scala, SQL, R |
| Dependencies | withDependencies() |
CLI with environments spec | Install on cluster |
Decision Flow
Spark-based code? → Databricks Connect (fastest)
└─ Python 3.12 missing? → Install it + databricks-connect
└─ Install fails? → Ask user (don't auto-switch modes)
Heavy/long-running (ML)? → Serverless Job (independent)
Need state across calls? → Interactive Cluster (list and ask which one to use)
Scala/R? → Interactive Cluster (list and ask which one to use)
How to Run Code
Read the reference file for your chosen mode before proceeding.
Databricks Connect (no MCP tool, run locally) → reference
python my_spark_script.py
Serverless Job → reference
execute_code(file_path="/path/to/script.py")
Interactive Cluster → reference
# Check for running clusters first (or use the one instructed)
list_compute(resource="clusters")
# Ask the customer which one to use
# Run code, reuse context_id for follow-up MCP call
result = execute_code(code="...", compute_type="cluster", cluster_id="...")
execute_code(code="...", context_id=result["context_id"], cluster_id=result["cluster_id"])
MCP Tools
| Tool | For | Purpose |
|---|---|---|
execute_code |
Serverless, Interactive | Run code remotely |
list_compute |
Interactive | List clusters, check status, auto-select running cluster |
manage_cluster |
Interactive | Create, start, terminate, delete. COSTLY: start takes 3-8 min—ask user |
manage_sql_warehouse |
SQL | Create, modify, delete SQL warehouses |
Related Skills
- databricks-synthetic-data-gen — Data generation using Spark + Faker
- databricks-jobs — Production job orchestration
- databricks-dbsql — SQL warehouse and AI functions