Databricks Applications — Python backends
First, confirm this skill is the right one. The default for new Databricks Apps is databricks-apps (AppKit — Node.js + TypeScript + React SDK). Load that skill first unless the user explicitly asks for a Python backend, is extending an existing Python app, or the team is Python-only. Everything below is the Python-backend alternative.
Critical Rules for Python apps (always follow)
- MUST confirm framework choice or use Python Framework Selection below
- MUST use SDK
Config() for authentication (never hardcode tokens)
- MUST use
app.yaml valueFrom for resources (never hardcode resource IDs)
- MUST use
dash-bootstrap-components for Dash app layout and styling
- MUST use
@st.cache_resource for Streamlit database connections
- MUST deploy Flask with Gunicorn, FastAPI with uvicorn (not dev servers)
Required Steps for Python apps
Copy this checklist and verify each item:
- [ ] Framework selected
- [ ] Auth strategy decided: app auth, user auth, or both
- [ ] App resources identified (SQL warehouse, Lakebase, serving endpoint, etc.)
- [ ] Backend data strategy decided (SQL warehouse, Lakebase, or SDK)
- [ ] Deployment method: CLI or DABs
Python Framework Selection
| Framework |
Best For |
app.yaml Command |
| FastAPI (default) |
Any Python backend by default — async APIs, auto-generated OpenAPI docs, JSON-serving apps |
["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"] |
| Flask |
Custom REST APIs, lightweight apps, webhooks |
["gunicorn", "app:app", "-w", "4", "-b", "0.0.0.0:8000"] |
| Dash |
Production dashboards, BI tools, complex interactivity |
["python", "app.py"] |
| Streamlit |
Rapid prototyping, data science apps, internal tools where the UI is a series of Python widgets |
["streamlit", "run", "app.py"] |
| Gradio |
ML demos, model interfaces, chat UIs |
["python", "app.py"] |
| Reflex |
Full-stack Python apps without JavaScript |
["reflex", "run", "--env", "prod"] |
Default: FastAPI. Reach for FastAPI unless the user explicitly asks for Streamlit-style widget prototyping (Streamlit), a heavy dashboard grid (Dash), or a Gradio-style ML demo. FastAPI pairs naturally with a JS/HTML frontend or a JSON-consuming caller — the same posture databricks-apps uses on the Node side.
Quick Reference
| Concept |
Details |
| Runtime |
Python 3.11, Ubuntu 22.04, 2 vCPU, 6 GB RAM |
| Pre-installed |
Dash 2.18.1, Streamlit 1.38.0, Gradio 4.44.0, Flask 3.0.3, FastAPI 0.115.0 |
| Auth (app) |
Service principal via Config() — auto-injected DATABRICKS_CLIENT_ID/DATABRICKS_CLIENT_SECRET |
| Auth (user) |
x-forwarded-access-token header — see references/1-authorization.md |
| Resources |
valueFrom in app.yaml — see references/2-app-resources.md |
| SDK / Foundation Models / Model Serving |
Use the databricks-python-sdk skill — same WorkspaceClient and OpenAI-compatible foundation-model patterns work inside a Databricks App. For UC model services (system.ai.*), see the model-service note in references/2-app-resources.md and the databricks-unity-gateway skill |
| AI Search (Vector Search) |
Use the databricks-vector-search skill: AISearchClient(workspace_url=..., service_principal_client_id=..., service_principal_client_secret=...) from DATABRICKS_HOST (prefix https:// if missing), DATABRICKS_CLIENT_ID, and DATABRICKS_CLIENT_SECRET. Create the client and call get_index() once at startup, then reuse the index object across requests. Add databricks-ai-search>=0.83 to the app's dependencies (e.g. requirements.txt). WorkspaceClient queries are capped at a few tens of QPS |
| Docs |
https://docs.databricks.com/dev-tools/databricks-apps/ |
Detailed Guides
Authorization: Use references/1-authorization.md when configuring app or user authorization — covers service principal auth, on-behalf-of user tokens, OAuth scopes, and per-framework code examples. (Keywords: OAuth, service principal, user auth, on-behalf-of, access token, scopes)
App resources: Use references/2-app-resources.md when connecting your app to Databricks resources — covers SQL warehouses, Lakebase, model serving, secrets, volumes, and the valueFrom pattern. (Keywords: resources, valueFrom, SQL warehouse, model serving, secrets, volumes, connections)
Frameworks: See references/3-frameworks.md for Databricks-specific patterns per framework — FastAPI (default), Flask, Dash, Streamlit, Gradio, Reflex — with auth integration and deployment commands. (Keywords: FastAPI, Flask, Dash, Streamlit, Gradio, Reflex, framework selection)
Deployment: Use references/4-deployment.md when deploying your app — covers Databricks CLI, Declarative Automation Bundles (DABs), app.yaml configuration, and post-deployment verification. (Keywords: deploy, CLI, DABs, asset bundles, app.yaml, logs)
Lakebase: Use references/5-lakebase.md when using Lakebase (PostgreSQL) as your app's data layer — covers auto-injected env vars, psycopg2/asyncpg patterns, and when to choose Lakebase vs SQL warehouse. (Keywords: Lakebase, PostgreSQL, psycopg2, asyncpg, transactional, PGHOST)
CLI commands: Use references/6-cli-approach.md for managing app lifecycle via CLI — covers creating, deploying, monitoring, and deleting apps. (Keywords: CLI, create app, deploy app, app logs)
Foundation Models / SDK / Model Serving: Use the databricks-python-sdk skill for the OpenAI-compatible foundation-model client, WorkspaceClient calls, and model-serving invocation — the same patterns apply inside a Databricks App. For AI Search (Vector Search) queries, use the databricks-vector-search skill instead (AISearchClient with the app's service principal; see Production and High QPS). The examples in this skill's examples/ folder (fm-minimal-chat.py, fm-parallel-calls.py, fm-structured-outputs.py, llm_config.py) show the App-side wiring only. For UC model services (system.ai.*, called on /ai-gateway/mlflow/v1), see the model-service note in references/2-app-resources.md and the databricks-unity-gateway skill.
Workflow
Determine the task type:
New app from scratch? → Load databricks-apps first (AppKit / Node). Only stay in this skill if the user explicitly asks for a Python backend.
Python-backend confirmed? → Python Framework Selection — default to FastAPI.
Setting up authorization? → Read references/1-authorization.md
Connecting to data/resources? → Read references/2-app-resources.md
Using Lakebase (PostgreSQL)? → Read references/5-lakebase.md
Deploying to Databricks? → Read references/4-deployment.md
Using CLI for app lifecycle? → Read references/6-cli-approach.md
Calling foundation model / LLM APIs or model-serving endpoints? → Load the databricks-python-sdk skill. This skill's examples/ folder shows only the App-side wiring on top of those SDK patterns.
Querying AI Search (Vector Search)? → Load the databricks-vector-search skill.
Follow the instructions in the relevant guide.
Core Architecture
All Python Databricks apps follow this pattern:
app-directory/
├── app.py # Main application (or framework-specific name)
├── models.py # Pydantic data models
├── backend.py # Data access layer
├── requirements.txt # Additional Python dependencies
├── app.yaml # Databricks Apps configuration
└── README.md
Backend Toggle Pattern
import os
from databricks.sdk.core import Config
USE_MOCK = os.getenv("USE_MOCK_BACKEND", "true").lower() == "true"
if USE_MOCK:
from backend_mock import MockBackend as Backend
else:
from backend_real import RealBackend as Backend
backend = Backend()
SQL Warehouse Connection (shared across all frameworks)
from databricks.sdk.core import Config
from databricks import sql
cfg = Config() # Auto-detects credentials from environment
conn = sql.connect(
server_hostname=cfg.host,
http_path=f"/sql/1.0/warehouses/{os.getenv('DATABRICKS_WAREHOUSE_ID')}",
credentials_provider=lambda: cfg.authenticate,
)
Pydantic Models
from pydantic import BaseModel, Field
from datetime import datetime
from enum import Enum
class Status(str, Enum):
ACTIVE = "active"
PENDING = "pending"
class EntityOut(BaseModel):
id: str
name: str
status: Status
created_at: datetime
class EntityIn(BaseModel):
name: str = Field(..., min_length=1)
status: Status = Status.PENDING
Common Issues
| Issue |
Solution |
| Connection exhausted |
Use @st.cache_resource (Streamlit) or connection pooling |
| Auth token not found |
Check x-forwarded-access-token header — only available when deployed, not locally |
| App won't start |
Check app.yaml command matches framework; check databricks apps logs <name> |
| Resource not accessible |
Add resource via UI, verify SP has permissions, use valueFrom in app.yaml |
| Import error on deploy |
Add missing packages to requirements.txt (pre-installed packages don't need listing) |
| Lakebase app crashes on start |
psycopg2/asyncpg are NOT pre-installed — MUST add to requirements.txt |
| Port conflict |
Apps must bind to DATABRICKS_APP_PORT env var (defaults to 8000). Never use 8080. Streamlit is auto-configured; for others, read the env var in code or use 8000 in app.yaml command |
| Streamlit: set_page_config error |
st.set_page_config() must be the first Streamlit command |
| Dash: unstyled layout |
Add dash-bootstrap-components; use dbc.themes.BOOTSTRAP |
| Slow queries |
Use Lakebase for transactional/low-latency; SQL warehouse for analytical queries |
Platform Constraints
| Constraint |
Details |
| Runtime |
Python 3.11, Ubuntu 22.04 LTS |
| Compute |
2 vCPUs, 6 GB memory (default) |
| Pre-installed frameworks |
Dash, Streamlit, Gradio, Flask, FastAPI, Shiny |
| Custom packages |
Add to requirements.txt in app root |
| Network |
Apps can reach Databricks APIs; external access depends on workspace config |
| User auth |
GA, on by default — no admin enablement required; declare user_api_scopes to add scopes |
Official Documentation
Related Skills
- databricks-apps — the default for new Databricks Apps (AppKit / Node / TypeScript + React); load it first unless a Python backend is explicitly required
- databricks-python-sdk —
WorkspaceClient, OpenAI-compatible foundation-model client, model-serving invocation; the same patterns work inside a Databricks App
- databricks-vector-search — AI Search (Vector Search) indexes and high-QPS queries with
AISearchClient
- databricks-lakebase — persistent PostgreSQL state (autoscaling managed PG with branching)
- databricks-model-serving — endpoint lifecycle for ML models an App calls
- databricks-dabs — deploying apps via DABs
1---2name: databricks-apps-python3description: "Python backend for Databricks Apps — FastAPI (default), Flask, Dash, Streamlit, Gradio, Reflex. **Default for a new Databricks App is `databricks-apps` (AppKit — Node/TypeScript/React) — reach for it first.** Use this skill only when the user asks for a Python backend, extends an existing Python app, or the team is Python-only. Covers OAuth auth, app resources, SQL warehouse and Lakebase connectivity, foundation-model / model-serving APIs (via `databricks-python-sdk`), AI Search (Vector Search) queries (via `databricks-vector-search`), and deployment via CLI or DABs."4compatibility: Requires databricks CLI (>= v1.0.0)5metadata:6 source: https://skillmd.com/skills/stanfish06/databricks-apps-python7 version: "0.1.0"8parent: databricks-core9---1011# Databricks Applications — Python backends1213> **First, confirm this skill is the right one.** The default for new Databricks Apps is **[databricks-apps](../databricks-apps/SKILL.md)** (AppKit — Node.js + TypeScript + React SDK). Load that skill first unless the user explicitly asks for a Python backend, is extending an existing Python app, or the team is Python-only. Everything below is the Python-backend alternative.1415## Critical Rules for Python apps (always follow)1617- **MUST** confirm framework choice or use [Python Framework Selection](#python-framework-selection) below18- **MUST** use SDK `Config()` for authentication (never hardcode tokens)19- **MUST** use `app.yaml` `valueFrom` for resources (never hardcode resource IDs)20- **MUST** use `dash-bootstrap-components` for Dash app layout and styling21- **MUST** use `@st.cache_resource` for Streamlit database connections22- **MUST** deploy Flask with Gunicorn, FastAPI with uvicorn (not dev servers)2324## Required Steps for Python apps2526Copy this checklist and verify each item:27```28- [ ] Framework selected29- [ ] Auth strategy decided: app auth, user auth, or both30- [ ] App resources identified (SQL warehouse, Lakebase, serving endpoint, etc.)31- [ ] Backend data strategy decided (SQL warehouse, Lakebase, or SDK)32- [ ] Deployment method: CLI or DABs33```3435---3637## Python Framework Selection3839| Framework | Best For | app.yaml Command |40|-----------|----------|------------------|41| **FastAPI** (default) | Any Python backend by default — async APIs, auto-generated OpenAPI docs, JSON-serving apps | `["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]` |42| **Flask** | Custom REST APIs, lightweight apps, webhooks | `["gunicorn", "app:app", "-w", "4", "-b", "0.0.0.0:8000"]` |43| **Dash** | Production dashboards, BI tools, complex interactivity | `["python", "app.py"]` |44| **Streamlit** | Rapid prototyping, data science apps, internal tools where the UI is a series of Python widgets | `["streamlit", "run", "app.py"]` |45| **Gradio** | ML demos, model interfaces, chat UIs | `["python", "app.py"]` |46| **Reflex** | Full-stack Python apps without JavaScript | `["reflex", "run", "--env", "prod"]` |4748**Default: FastAPI.** Reach for FastAPI unless the user explicitly asks for Streamlit-style widget prototyping (Streamlit), a heavy dashboard grid (Dash), or a Gradio-style ML demo. FastAPI pairs naturally with a JS/HTML frontend or a JSON-consuming caller — the same posture `databricks-apps` uses on the Node side.4950---5152## Quick Reference5354| Concept | Details |55|---------|---------|56| **Runtime** | Python 3.11, Ubuntu 22.04, 2 vCPU, 6 GB RAM |57| **Pre-installed** | Dash 2.18.1, Streamlit 1.38.0, Gradio 4.44.0, Flask 3.0.3, FastAPI 0.115.0 |58| **Auth (app)** | Service principal via `Config()` — auto-injected `DATABRICKS_CLIENT_ID`/`DATABRICKS_CLIENT_SECRET` |59| **Auth (user)** | `x-forwarded-access-token` header — see [references/1-authorization.md](references/1-authorization.md) |60| **Resources** | `valueFrom` in app.yaml — see [references/2-app-resources.md](references/2-app-resources.md) |61| **SDK / Foundation Models / Model Serving** | Use the `databricks-python-sdk` skill — same `WorkspaceClient` and OpenAI-compatible foundation-model patterns work inside a Databricks App. For UC model services (`system.ai.*`), see the model-service note in [references/2-app-resources.md](references/2-app-resources.md) and the `databricks-unity-gateway` skill |62| **AI Search (Vector Search)** | Use the `databricks-vector-search` skill: `AISearchClient(workspace_url=..., service_principal_client_id=..., service_principal_client_secret=...)` from `DATABRICKS_HOST` (prefix `https://` if missing), `DATABRICKS_CLIENT_ID`, and `DATABRICKS_CLIENT_SECRET`. Create the client and call `get_index()` once at startup, then reuse the index object across requests. Add `databricks-ai-search>=0.83` to the app's dependencies (e.g. `requirements.txt`). `WorkspaceClient` queries are capped at a few tens of QPS |63| **Docs** | https://docs.databricks.com/dev-tools/databricks-apps/ |6465---6667## Detailed Guides6869**Authorization**: Use [references/1-authorization.md](references/1-authorization.md) when configuring app or user authorization — covers service principal auth, on-behalf-of user tokens, OAuth scopes, and per-framework code examples. (Keywords: OAuth, service principal, user auth, on-behalf-of, access token, scopes)7071**App resources**: Use [references/2-app-resources.md](references/2-app-resources.md) when connecting your app to Databricks resources — covers SQL warehouses, Lakebase, model serving, secrets, volumes, and the `valueFrom` pattern. (Keywords: resources, valueFrom, SQL warehouse, model serving, secrets, volumes, connections)7273**Frameworks**: See [references/3-frameworks.md](references/3-frameworks.md) for Databricks-specific patterns per framework — FastAPI (default), Flask, Dash, Streamlit, Gradio, Reflex — with auth integration and deployment commands. (Keywords: FastAPI, Flask, Dash, Streamlit, Gradio, Reflex, framework selection)7475**Deployment**: Use [references/4-deployment.md](references/4-deployment.md) when deploying your app — covers Databricks CLI, Declarative Automation Bundles (DABs), app.yaml configuration, and post-deployment verification. (Keywords: deploy, CLI, DABs, asset bundles, app.yaml, logs)7677**Lakebase**: Use [references/5-lakebase.md](references/5-lakebase.md) when using Lakebase (PostgreSQL) as your app's data layer — covers auto-injected env vars, psycopg2/asyncpg patterns, and when to choose Lakebase vs SQL warehouse. (Keywords: Lakebase, PostgreSQL, psycopg2, asyncpg, transactional, PGHOST)7879**CLI commands**: Use [references/6-cli-approach.md](references/6-cli-approach.md) for managing app lifecycle via CLI — covers creating, deploying, monitoring, and deleting apps. (Keywords: CLI, create app, deploy app, app logs)8081**Foundation Models / SDK / Model Serving**: Use the **[databricks-python-sdk](../databricks-python-sdk/SKILL.md)** skill for the OpenAI-compatible foundation-model client, `WorkspaceClient` calls, and model-serving invocation — the same patterns apply inside a Databricks App. For AI Search (Vector Search) queries, use the **[databricks-vector-search](../databricks-vector-search/SKILL.md)** skill instead (`AISearchClient` with the app's service principal; see [Production and High QPS](../databricks-vector-search/SKILL.md#production-and-high-qps)). The examples in this skill's `examples/` folder (`fm-minimal-chat.py`, `fm-parallel-calls.py`, `fm-structured-outputs.py`, `llm_config.py`) show the App-side wiring only. For **UC model services** (`system.ai.*`, called on `/ai-gateway/mlflow/v1`), see the model-service note in [references/2-app-resources.md](references/2-app-resources.md) and the **`databricks-unity-gateway`** skill.8283---8485## Workflow86871. Determine the task type:8889 **New app from scratch?** → Load **[databricks-apps](../databricks-apps/SKILL.md)** first (AppKit / Node). Only stay in this skill if the user explicitly asks for a Python backend.90 **Python-backend confirmed?** → [Python Framework Selection](#python-framework-selection) — default to FastAPI.91 **Setting up authorization?** → Read [references/1-authorization.md](references/1-authorization.md)92 **Connecting to data/resources?** → Read [references/2-app-resources.md](references/2-app-resources.md)93 **Using Lakebase (PostgreSQL)?** → Read [references/5-lakebase.md](references/5-lakebase.md)94 **Deploying to Databricks?** → Read [references/4-deployment.md](references/4-deployment.md)95 **Using CLI for app lifecycle?** → Read [references/6-cli-approach.md](references/6-cli-approach.md)96 **Calling foundation model / LLM APIs or model-serving endpoints?** → Load the **[databricks-python-sdk](../databricks-python-sdk/SKILL.md)** skill. This skill's `examples/` folder shows only the App-side wiring on top of those SDK patterns.97 **Querying AI Search (Vector Search)?** → Load the **[databricks-vector-search](../databricks-vector-search/SKILL.md)** skill.98992. Follow the instructions in the relevant guide.100101---102103## Core Architecture104105All Python Databricks apps follow this pattern:106107```108app-directory/109├── app.py # Main application (or framework-specific name)110├── models.py # Pydantic data models111├── backend.py # Data access layer112├── requirements.txt # Additional Python dependencies113├── app.yaml # Databricks Apps configuration114└── README.md115```116117### Backend Toggle Pattern118119```python120import os121from databricks.sdk.core import Config122123USE_MOCK = os.getenv("USE_MOCK_BACKEND", "true").lower() == "true"124125if USE_MOCK:126 from backend_mock import MockBackend as Backend127else:128 from backend_real import RealBackend as Backend129130backend = Backend()131```132133### SQL Warehouse Connection (shared across all frameworks)134135```python136from databricks.sdk.core import Config137from databricks import sql138139cfg = Config() # Auto-detects credentials from environment140conn = sql.connect(141 server_hostname=cfg.host,142 http_path=f"/sql/1.0/warehouses/{os.getenv('DATABRICKS_WAREHOUSE_ID')}",143 credentials_provider=lambda: cfg.authenticate,144)145```146147### Pydantic Models148149```python150from pydantic import BaseModel, Field151from datetime import datetime152from enum import Enum153154class Status(str, Enum):155 ACTIVE = "active"156 PENDING = "pending"157158class EntityOut(BaseModel):159 id: str160 name: str161 status: Status162 created_at: datetime163164class EntityIn(BaseModel):165 name: str = Field(..., min_length=1)166 status: Status = Status.PENDING167```168169---170171## Common Issues172173| Issue | Solution |174|-------|----------|175| **Connection exhausted** | Use `@st.cache_resource` (Streamlit) or connection pooling |176| **Auth token not found** | Check `x-forwarded-access-token` header — only available when deployed, not locally |177| **App won't start** | Check `app.yaml` command matches framework; check `databricks apps logs <name>` |178| **Resource not accessible** | Add resource via UI, verify SP has permissions, use `valueFrom` in app.yaml |179| **Import error on deploy** | Add missing packages to `requirements.txt` (pre-installed packages don't need listing) |180| **Lakebase app crashes on start** | `psycopg2`/`asyncpg` are NOT pre-installed — MUST add to `requirements.txt` |181| **Port conflict** | Apps must bind to `DATABRICKS_APP_PORT` env var (defaults to 8000). Never use 8080. Streamlit is auto-configured; for others, read the env var in code or use 8000 in app.yaml command |182| **Streamlit: set_page_config error** | `st.set_page_config()` must be the first Streamlit command |183| **Dash: unstyled layout** | Add `dash-bootstrap-components`; use `dbc.themes.BOOTSTRAP` |184| **Slow queries** | Use Lakebase for transactional/low-latency; SQL warehouse for analytical queries |185186---187188## Platform Constraints189190| Constraint | Details |191|------------|---------|192| **Runtime** | Python 3.11, Ubuntu 22.04 LTS |193| **Compute** | 2 vCPUs, 6 GB memory (default) |194| **Pre-installed frameworks** | Dash, Streamlit, Gradio, Flask, FastAPI, Shiny |195| **Custom packages** | Add to `requirements.txt` in app root |196| **Network** | Apps can reach Databricks APIs; external access depends on workspace config |197| **User auth** | GA, on by default — no admin enablement required; declare `user_api_scopes` to add scopes |198199---200201## Official Documentation202203- **[Databricks Apps Overview](https://docs.databricks.com/dev-tools/databricks-apps/)** — main docs hub204- **[Authorization](https://docs.databricks.com/dev-tools/databricks-apps/auth)** — app auth and user auth205- **[Resources](https://docs.databricks.com/dev-tools/databricks-apps/resources)** — SQL warehouse, Lakebase, serving, secrets206- **[app.yaml Reference](https://docs.databricks.com/dev-tools/databricks-apps/app-runtime)** — command and env config207- **[System Environment](https://docs.databricks.com/dev-tools/databricks-apps/system-env)** — pre-installed packages, runtime details208209## Related Skills210211- **[databricks-apps](../databricks-apps/SKILL.md)** — the default for new Databricks Apps (AppKit / Node / TypeScript + React); load it first unless a Python backend is explicitly required212- **[databricks-python-sdk](../databricks-python-sdk/SKILL.md)** — `WorkspaceClient`, OpenAI-compatible foundation-model client, model-serving invocation; the same patterns work inside a Databricks App213- **[databricks-vector-search](../databricks-vector-search/SKILL.md)** — AI Search (Vector Search) indexes and high-QPS queries with `AISearchClient`214- **[databricks-lakebase](../databricks-lakebase/SKILL.md)** — persistent PostgreSQL state (autoscaling managed PG with branching)215- **[databricks-model-serving](../databricks-model-serving/SKILL.md)** — endpoint lifecycle for ML models an App calls216- **databricks-dabs** — deploying apps via DABs