Databricks Isv Databricks Connect

Databricks Connect (ISV)

Use this skill when implementing or testing Databricks Connect (Python) for PWAF-compliant partner integrations: remote Spark execution from external apps using databricks-connect and DatabricksSession.

Reference

• Full patterns: authentication.md – PAT, OAuth M2M, OAuth U2M (external-browser, localhost, token-env), serverless and classic compute, User-Agent, version compatibility, troubleshooting.

Golden snippets (copy-paste accurate)

Serverless PAT – Config and session with .userAgent():

from databricks.connect import DatabricksSession
from databricks.sdk.config import Config

config = Config(
    host=host_url,
    token=token,
    serverless_compute_id="auto",
)
session = (
    DatabricksSession.builder
        .sdkConfig(config)
        .userAgent("YourCompany_YourProduct/1.0.0")
        .getOrCreate()
)

OAuth M2M – always set auth_type:

config = Config(
    host=host_url,
    client_id=client_id,
    client_secret=client_secret,
    auth_type="oauth-m2m",   # required
    serverless_compute_id="auto",
)
session = (
    DatabricksSession.builder
        .sdkConfig(config)
        .userAgent("YourCompany_YourProduct/1.0.0")
        .getOrCreate()
)

U2M – get token first (e.g. external-browser), then Config with token: Same as PAT but token=access_token from browser flow. For external-browser, unset DATABRICKS_CLIENT_ID/DATABRICKS_CLIENT_SECRET before calling SDK.

Requirements

• User-Agent (required): Use .userAgent("<isv>_<product>/<version>") on DatabricksSession.builder (PWAF-recommended). Do not expose as user config. See PWAF Databricks Connect telemetry.
• Auth types: Support PAT, OAuth M2M, and OAuth U2M. One connection = one auth type; do not set both PAT and M2M (or U2M) vars in the same process.
• OAuth M2M: Always pass auth_type="oauth-m2m" to Config(...) when using client_id/client_secret so the SDK does not run default credential resolution.

Compute options

• Serverless (recommended): DATABRICKS_SERVERLESS_COMPUTE_ID=auto or Config(..., serverless_compute_id="auto"). No cluster to manage. Some Connect versions do not yet support serverless; in that case use classic.
• Classic: DATABRICKS_CLUSTER_ID or CLASSIC_COMPUTE_HTTP_PATH (cluster ID = last path segment, e.g. sql/protocolv1/o/<workspace_id>/<cluster_id>).
• Rule: Do not set both serverless and classic in the same env; Config errors with "Can't set both cluster id and serverless_compute_id".

U2M with Connect

Obtain an access token first, then pass it to Config(host=..., token=access_token, ...) for DatabricksSession. Same three flows as REST/SQL U2M:

Important: M2M service principal client_id is not valid for browser U2M. For external-browser, do not pass client_id. For localhost, use a separate OAuth custom app.

Version compatibility

• databricks-connect version must match the Databricks Runtime (DBR) on compute (e.g. DBR 18 → databricks-connect==18.0.*).
• Do not install PySpark separately; databricks-connect supplies the matching Spark stack.

Auth isolation

Run each test or script with a clean environment: only the vars for the chosen auth type and compute. Use env -i plus explicit vars when running tests so PAT, M2M, and U2M do not mix (avoids "more than one authorization method configured").

Validation

• Build DatabricksSession with the chosen auth and compute; run a simple read (e.g. spark.table("samples.nyctaxi.trips").count()) to verify connection and permissions.