TS Profile Databricks

Databricks Profile Setup

Manage Databricks connection profiles stored in ~/.claude/databricks-profiles.json.

Ask one question at a time for dependent decisions (credential flows are mostly sequential). Batch independent questions when possible — e.g. profile name + host can be collected together.

Prerequisites

Before running this skill, complete these steps in your Databricks environment.

1. Install the Databricks CLI

Verify: databricks --version

2. Know your workspace URL

Your workspace URL looks like https://dbc-abc123.cloud.databricks.com (AWS) or https://adb-1234567890.1.azuredatabricks.net (Azure). Find it by logging into your workspace and copying the URL from the browser address bar.

Important: This skill needs a workspace-level URL, not the account-level https://accounts.cloud.databricks.com. Account-level profiles are for administration only — they cannot access catalogs, run SQL, or list warehouses.

3. Prepare your auth credentials

Choose one of the following. If unsure, start with Service Principal for automation or PAT for personal use.

Option A — Service Principal (recommended for automation)

A Service Principal is a machine identity that doesn't expire when someone leaves. Create one before running this skill:

1. Create the Service Principal: Account Console → Settings → Service principals → Add service principal https://docs.databricks.com/en/admin/users-groups/service-principals.html

2. Generate an OAuth secret: Service principal → Settings → Generate secret Copy the client ID and secret — you'll need both. https://docs.databricks.com/en/dev-tools/authentication-oauth.html

3. Add the SP to your workspace: Account Console → Workspaces → {your workspace} → Permissions → Add service principal https://docs.databricks.com/en/admin/users-groups/service-principals.html#assign-a-service-principal-to-a-workspace

4. Grant catalog and warehouse access (run in a SQL editor as admin):

   GRANT USE CATALOG ON CATALOG {catalog_name} TO `{service_principal_application_id}`;
   GRANT USE SCHEMA ON SCHEMA {catalog_name}.{schema_name} TO `{service_principal_application_id}`;
   

Option B — Personal Access Token (PAT)

Good for personal use and quick setup. Tokens expire (default 90 days).

1. Open your Databricks workspace in a browser
2. Click your username (top-right) → Settings
3. Go to Developer → Access Tokens
4. Click Generate New Token — set a description and expiry
5. Copy the token value (you won't see it again)

https://docs.databricks.com/en/dev-tools/auth/pat.html

Option C — Existing Databricks CLI profile

If you've already configured a workspace-level profile via databricks auth login, you can reuse it directly. Run databricks auth profiles to see your profiles.

Account-level profiles (host = accounts.cloud.databricks.com) won't work for workspace operations. You'll need to create a workspace-level profile first:

databricks auth login --host https://your-workspace.cloud.databricks.com

4. Know your SQL Warehouse HTTP path (optional)

Required for skills that run SQL statements. Find it in your workspace: Workspace → SQL Warehouses → {warehouse} → Connection Details → HTTP Path

Example: /sql/1.0/warehouses/abc123def456

Step 0 — Overview

On skill invocation, display this introduction before reading the profiles file:

ts-profile-databricks — manage your Databricks connection profiles.

Profiles store your Databricks workspace URL, auth credentials, and default catalog/schema/warehouse. Credentials are kept securely in your OS keychain (macOS Keychain, Windows Credential Manager, or Linux Secret Service) — never in the profile file or this conversation.

Prerequisites: Databricks CLI installed, workspace URL, and auth credentials ready (Service Principal, PAT, or existing CLI profile). See the Prerequisites section if you need to set these up first.

Actions: L List A Add U Update D Delete T Test

Reading profiles…

Then proceed to Entry Point.

Entry Point

Read ~/.claude/databricks-profiles.json.

If no profiles file or empty profiles array: go directly to Add.

If profiles exist: show the menu.

Databricks Profiles

  1. {name}  —  {auth_type_label}  —  {host}
  2. {name}  —  {auth_type_label}  —  {host}
  ...

  L  List profiles       (full details of all profiles)
  A  Add a new profile
  U  Update a profile
  D  Delete a profile
  T  Test a profile
  Q  Quit

Enter L / A / U / D / T / Q:

For auth_type_label display: oauth-m2m → service principal, pat → personal access token, databricks-cli → CLI profile.

L — List

Show each profile with full details. For each profile:

Service principal or PAT:

Profile: {name}
  Host:      {host}
  Auth:      {auth_type_label}
  {if oauth-m2m: Client ID: {client_id}}
  Catalog:   {default_catalog}
  Schema:    {default_schema}
  Warehouse: {sql_warehouse_http_path}
  Env var:   {secret_env}
  Status:    {SET — credential loaded | NOT SET — run 'source ~/.zshenv' (macOS/Linux) or restart terminal (Windows), or re-add credential}

CLI profile:

Profile: {name}
  Host:       {host}
  Auth:       CLI profile ({dbx_cli_profile})
  Catalog:    {default_catalog}
  Schema:     {default_schema}
  Warehouse:  {sql_warehouse_http_path}

To check env var status: read os.environ.get("{secret_env}", "") — non-empty = SET.

After displaying all profiles, return to the menu.

A — Add

A1 — Check Databricks CLI

Run: databricks --version 2>&1

If not found:

Databricks CLI is not installed.

Install it from:
  macOS:   brew tap databricks/tap && brew install databricks
  pip:     pip install databricks-cli
  Other:   https://docs.databricks.com/en/dev-tools/cli/install.html

Once installed, run `databricks --version` to verify, then re-run /ts-profile-databricks.

Stop here.

A2 — Choose Auth Method

Auth method:

  1  Service principal  — OAuth M2M; best for shared/automated use, doesn't expire with people (recommended)
  2  Personal access token (PAT)  — user-scoped token; quick to set up, expires (default 90 days)
  3  Existing CLI profile  — reuse a workspace-level profile already in ~/.databrickscfg

Enter 1, 2, or 3:

If the user is unsure: recommend 1 (Service Principal) for team or production use, 2 (PAT) for personal exploration, 3 only if they've already run databricks auth login for their workspace.

Path A — Service Principal (OAuth M2M)

A-SP1 — Collect Workspace Details

Ask one at a time:

Databricks workspace URL (e.g. https://dbc-abc123.cloud.databricks.com):

Strip trailing slash. Store as {host}.

Service Principal client ID (Application ID from the SP settings):

Store as {client_id}.

Profile name: [Production]

Default to Production. Store as {profile_name}.

A-SP2 — Save Profile and Store Client Secret

ts profiles add \
  --platform databricks \
  --name "{profile_name}" \
  --auth-type oauth-m2m \
  --field host={host} \
  --field client_id={client_id}

Parse the JSON output. It contains slug, keychain_store_commands, keychain_verify_commands, zshenv_line, windows_env_commands, and keychain_service.

Never accept the credential in this conversation.

Show the user the keychain store command for their platform from the keychain_store_commands in the output, replacing VALUE with YOUR_CLIENT_SECRET_HERE. Tell them to run it in their own terminal.

After the user confirms, show the verify command from keychain_verify_commands.

Stop if verification fails.

A-SP3 — Update Shell Profile

macOS / Linux: Read ~/.zshenv, upsert the zshenv_line from the output (replace an existing line for the same env var, or append if not present), write back. Tell the user to run source ~/.zshenv and wait for confirmation.

Windows: Show the windows_env_commands from the output for the user to run in PowerShell. Wait for confirmation.

A-SP4 — Verify CLI Auth (no secret on disk)

The Databricks CLI reads Service Principal credentials straight from the environment, so nothing needs to be written to ~/.databrickscfg and the secret never leaves the keychain. ~/.zshenv (step A-SP3) already exports {secret_env} from the keychain, so confirm auth works:

DATABRICKS_HOST="{host}" \
DATABRICKS_CLIENT_ID="{client_id}" \
DATABRICKS_CLIENT_SECRET="${secret_env}" \
  databricks current-user me -o json

A JSON body naming the Service Principal means auth is working. Live-verified 2026-08-26 against a real SP on Databricks CLI v1.13.0.

If it reports an empty secret, the user has not run source ~/.zshenv yet.

Why no ~/.databrickscfg section. This skill previously wrote client_secret there in plaintext, on the stated grounds that the CLI "does not support shell expansion or keychain lookups." The second half is true and the conclusion does not follow — the CLI supports DATABRICKS_HOST / DATABRICKS_CLIENT_ID / DATABRICKS_CLIENT_SECRET, which is a keychain-backed path that keeps the credential off disk entirely. Writing it was a direct violation of security.md ("Credentials are never stored in files"), and chmod 600 mitigates but does not fix that — a long-lived OAuth secret in a readable file is still a second copy outside the credential store, surviving backups and rsync.

Only if the user specifically needs a named CLI profile (e.g. to pass --profile {dbx_profile} or set DATABRICKS_CONFIG_PROFILE for agents/databricks/deploy.sh), tell them how to add it themselves and name the trade-off — do not write the secret to the file on their behalf:

Adding a named profile means storing the OAuth secret in ~/.databrickscfg in
plaintext. The env-var path above needs no such copy. If you still want the
named profile, add this yourself and keep the file at chmod 600:

  [{dbx_profile}]
  host      = {host}
  client_id = {client_id}
  client_secret = <your secret>
  auth_type = oauth-m2m

Note that deploy.sh only passes --profile when DATABRICKS_CONFIG_PROFILE is set; with it unset, the CLI picks up the env vars above and no profile is needed.

A-SP5 — Collect Workspace Defaults

Default catalog (e.g. main): [main]

Default to main. Store as {default_catalog}.

Default schema (e.g. default): [default]

Default to default. Store as {default_schema}.

SQL warehouse HTTP path (from Warehouse → Connection Details, e.g. /sql/1.0/warehouses/abc123):

Store as {sql_warehouse_http_path}. This is optional — if the user doesn't have one yet, store empty string.

If the user is unsure where to find this:

To find the HTTP path:
  1. Open your Databricks workspace in a browser
  2. Go to SQL Warehouses (in the left sidebar)
  3. Click on a warehouse
  4. Go to the Connection Details tab
  5. Copy the HTTP Path value

You can also skip this for now and add it later with U → Update.

Update the profile with workspace defaults:

ts profiles update --platform databricks --name "{profile_name}" \
  --field default_catalog={default_catalog} \
  --field default_schema={default_schema} \
  --field sql_warehouse_http_path={sql_warehouse_http_path}

Path B — Personal Access Token (PAT)

A-PAT1 — Collect Workspace Details

Ask one at a time:

Databricks workspace URL (e.g. https://dbc-abc123.cloud.databricks.com):

Strip trailing slash. Store as {host}.

Profile name: [Production]

Default to Production. Store as {profile_name}.

A-PAT2 — Save Profile and Store Token

ts profiles add \
  --platform databricks \
  --name "{profile_name}" \
  --auth-type pat \
  --field host={host}

Parse the JSON output for slug, keychain_store_commands, keychain_verify_commands, zshenv_line, windows_env_commands, keychain_service, and env_var.

To get your token:
  1. Open your Databricks workspace in a browser
  2. Click your username in the top-right → Settings
  3. Go to Developer → Access Tokens
  4. Click Generate New Token, set a description and expiry
  5. Copy the token value (you won't see it again)

Show the user the keychain store command for their platform from the keychain_store_commands in the output, replacing VALUE with YOUR_TOKEN_HERE. Tell them to run it in their own terminal.

After the user confirms, show the verify command from keychain_verify_commands.

Stop if verification fails.

A-PAT3 — Update Shell Profile

Same as A-SP3 — upsert the zshenv_line from the output.

A-PAT4 — Verify CLI Auth (no token on disk)

Same principle as A-SP4 — the CLI reads a PAT from the environment, so the token stays in the keychain and nothing is written to ~/.databrickscfg:

DATABRICKS_HOST="{host}" DATABRICKS_TOKEN="${secret_env}" \
  databricks current-user me -o json

An empty token means the user has not run source ~/.zshenv yet.

If the user specifically needs a named CLI profile, apply the same rule as A-SP4: tell them what it costs (the token in plaintext at chmod 600) and let them add it themselves — do not write the credential to the file for them.

A-PAT5 — Collect Workspace Defaults

Same as A-SP5 — collect default catalog, default schema, SQL warehouse HTTP path, and update the profile with ts profiles update.

Path C — Existing Databricks CLI Profile

A-CLI1 — List Available CLI Profiles

Run: databricks auth profiles 2>&1

Parse the profile names from the output and display them numbered:

Available Databricks CLI profiles:

  1  {profile_name_1}  —  {host_1}
  2  {profile_name_2}  —  {host_2}
  ...

Enter a number:

Store the selected profile name as {dbx_cli_profile}. Extract its host from the output as {host}.

A-CLI2 — Validate and Test CLI Profile

Account-level check: If the host is accounts.cloud.databricks.com (or contains accounts.), warn the user:

⚠ This is an account-level profile — it can manage users and workspaces but
cannot access catalogs, run SQL, or list warehouses.

You need a workspace-level profile instead. Create one with:

  databricks auth login --host https://your-workspace.cloud.databricks.com

Then re-run /ts-profile-databricks and select the new profile.

Stop here — do not proceed with an account-level profile.

Auth test: Run: databricks auth describe --profile {dbx_cli_profile} 2>&1

If it shows auth_type and host, the profile is usable. Show the auth details.

If it shows authentication errors, warn the user:

This CLI profile has authentication issues. You may need to re-authenticate:

  databricks auth login --profile {dbx_cli_profile}

Try re-authenticating, then let me know when done.

A-CLI3 — Collect Details and Save Profile

Profile name for this connection: [Production]

Default to Production. Store as {profile_name}.

ts profiles add \
  --platform databricks \
  --name "{profile_name}" \
  --auth-type databricks-cli \
  --field host={host} \
  --field dbx_profile={dbx_cli_profile}

CLI auth does not use a keychain entry or env var.

Then collect workspace defaults (same as A-SP5) and update with ts profiles update.

Test and Confirm

Run the Test flow for this profile.

On success:

Databricks profile '{profile_name}' configured and verified.

Return to menu (or exit if this was the first-run flow).

U — Update

Show numbered profile list and ask:

Which profile would you like to update? (enter number):

Then show what can be changed:

What would you like to update?

  1  Workspace URL
  2  Refresh credential  (update stored secret or token — same auth method)
  3  Change auth method  (switch between service principal / PAT / CLI profile)
  4  Default catalog
  5  Default schema
  6  SQL warehouse HTTP path

Enter 1–6:

U1 — Update Workspace URL

New workspace URL: [{current_host}]

ts profiles update --platform databricks --name "{profile_name}" --field host={new_host}

If — and only if — the user opted into a named CLI profile (see A-SP4), also update host in ~/.databrickscfg under the [{dbx_profile}] section if the profile uses SP or PAT auth.

Confirm: URL updated.

U2 — Refresh Credential

Service principal:

Detect platform, then ask the user to run in their own terminal.

The keychain service is databricks-{slug} and the account is the client_id.

macOS (Darwin):

Run this in your terminal to update the client secret:

  security delete-generic-password -s "databricks-{slug}" -a "{client_id}"
  security add-generic-password \
    -s "databricks-{slug}" \
    -a "{client_id}" \
    -w "YOUR_NEW_SECRET_HERE"

Let me know when done.

Windows (PowerShell):

Run this in PowerShell to update the client secret:

  python -c "import keyring; keyring.delete_password('databricks-{slug}', '{client_id}')"
  python -c "import keyring; keyring.set_password('databricks-{slug}', '{client_id}', 'YOUR_NEW_SECRET_HERE')"

Let me know when done.

Linux:

Run this in your terminal to update the client secret:

  python3 -c "import keyring; keyring.delete_password('databricks-{slug}', '{client_id}')"
  python3 -c "import keyring; keyring.set_password('databricks-{slug}', '{client_id}', 'YOUR_NEW_SECRET_HERE')"

Let me know when done.

After confirmation, verify (re-use the platform-specific verification block from the Add flow).

macOS / Linux: Tell the user to run source ~/.zshenv.

After sourcing, the env-var path picks the new secret up with no further step. Only if the user opted into a named CLI profile (A-SP4) does client_secret in ~/.databrickscfg under [{dbx_profile}] also need updating — the keychain entry is always the authoritative copy.

PAT: Same pattern but with account name "token", and updating the token field in ~/.databrickscfg.

CLI profile: Show:

CLI profiles are managed by the Databricks CLI. Run:

  databricks auth login --profile {dbx_cli_profile}

to re-authenticate, then let me know when done.

U3 — Change Auth Method

Run the full credential setup section of Add for the chosen method. ts profiles add will replace the existing profile entry.

Clean up the old auth:

• Delete old Keychain entry
• Remove old export line from ~/.zshenv (macOS/Linux)
• Remove or replace an old ~/.databrickscfg section, if the user had opted into one
• Profile JSON is already updated by ts profiles add

U4–U6 — Simple Field Updates (Catalog, Schema, Warehouse)

New {field}: [{current_value}]

ts profiles update --platform databricks --name "{profile_name}" --field {field_key}={new_value}

Confirm: {Field} updated.

D — Delete

Show numbered profile list and ask:

Which profile would you like to delete? (enter number):

Confirm:

Delete profile '{name}'?
This will remove it from the profile file{, the OS credential store, ~/.zshenv, and ~/.databrickscfg}.

Y / N:

If confirmed:

1. Remove profile:

ts profiles remove --platform databricks --name "{profile_name}"

Parse the JSON output for keychain_service and env_var_to_remove.

2. If SP or PAT auth — remove credential store entry:

   macOS:

   security delete-generic-password -s "{keychain_service}" -a "{account_name}"
   

   Where {account_name} is {client_id} for SP or "token" for PAT.

   Windows / Linux:

   import keyring
   try:
       keyring.delete_password("{keychain_service}", "{account_name}")
   except Exception:
       pass
   

   If not found, continue silently.

3. If SP or PAT auth — remove export line from shell profile (macOS/Linux only) — read ~/.zshenv, filter out any line that exports {env_var_to_remove}, write back.

4. If the user had opted into a named CLI profile — remove the ~/.databrickscfg section — read the file, remove the [{dbx_profile}] section, write back. Preserve all other sections. Skip when absent (the default: A-SP4/A-PAT4 write no section).

5. Tell the user:

Profile '{name}' deleted.

macOS / Linux (if SP or PAT): also tell the user:

Run this in your terminal to apply the ~/.zshenv change:

  source ~/.zshenv

Windows: no shell profile reload needed.

T — Test

Show numbered profile list (if more than one) and ask which to test. If only one, confirm and test it directly.

Test Step 1 — Auth Check

databricks auth describe --profile {dbx_profile} 2>&1

Check that the output shows a valid auth_type and no authentication errors.

On auth failure: Authentication failed — check credentials. Run U → Refresh credential.

Test Step 2 — Workspace Connectivity

databricks clusters list --profile {dbx_profile} -o json 2>&1

This verifies the profile can reach the workspace and authenticate. The output doesn't matter — a successful response (even empty list) confirms connectivity.

On failure: show the error and suggest checking the workspace URL or credentials.

Test Step 3 — Catalog Access (if default_catalog is set)

databricks catalogs get {default_catalog} --profile {dbx_profile} -o json 2>&1

On failure: Catalog '{default_catalog}' not accessible — check the catalog name and permissions.

Test Step 4 — SQL Warehouse (if sql_warehouse_http_path is set)

databricks warehouses list --profile {dbx_profile} -o json 2>&1

Parse the output to verify a warehouse matching {sql_warehouse_http_path} exists and is accessible.

On success:

Profile '{name}' — connection verified.

  Auth:      ✓
  Workspace: ✓
  {if tested: Catalog:   ✓ ({default_catalog})}
  {if tested: Warehouse: ✓}

Return to menu.

Error Handling

Technical Reference — For Use by Other Skills

Reading a Databricks Profile

import json, os

profiles_path = os.path.expanduser("~/.claude/databricks-profiles.json")
with open(profiles_path) as f:
    profiles = json.load(f)

profile = next(p for p in profiles if p["name"] == "{profile_name}")
dbx_profile = profile["dbx_profile"]
catalog = profile.get("default_catalog", "main")
schema = profile.get("default_schema", "default")
warehouse_path = profile.get("sql_warehouse_http_path", "")

Running Databricks CLI Commands

databricks clusters list --profile {dbx_profile} -o json
databricks catalogs list --profile {dbx_profile} -o json
databricks schemas list {catalog} --profile {dbx_profile} -o json
databricks tables list {catalog}.{schema} --profile {dbx_profile} -o json

Executing SQL via Databricks CLI

databricks api post /api/2.0/sql/statements \
  --profile {dbx_profile} \
  --json '{
    "warehouse_id": "{warehouse_id}",
    "statement": "SELECT 1",
    "wait_timeout": "30s"
  }'

The warehouse_id is extracted from sql_warehouse_http_path — it is the last path segment (e.g. /sql/1.0/warehouses/abc123 → abc123).

Python SDK Connection

import os
from databricks.sdk import WorkspaceClient

w = WorkspaceClient(
    host=profile["host"],
    client_id=profile.get("client_id"),
    client_secret=os.environ.get(profile.get("secret_env", ""), ""),
)

for c in w.catalogs.list():
    print(c.name)

Extracting Warehouse ID from HTTP Path

warehouse_id = profile["sql_warehouse_http_path"].rstrip("/").split("/")[-1]

Changelog