gabrielmoreira Databricks Bundle Medic
The deploy + infrastructure spine of the pack. Databricks Asset Bundles (DAB) is
immature tooling — the replacement for the deprecated dbx, with a moving bug list
across CLI versions — and the infrastructure operations around a deploy (encryption
keys, networking, workspace config) bite production platform teams hardest. This skill
turns the five worst deploy-time foot-guns into deterministic, reversible operations.
Overview
Five pains, all from the pack's deploy-ops research:
Asset Bundles (DAB). D4 — databricks bundle bind does NOT yet support UC catalogs
or external locations (databricks/cli#4842), so existing UC resources cannot come under
bundle management without hand-editing terraform.tfstate (hostile) or
destroy-and-recreate (impossible with dependent tables). D5 — "unexpected EOF reading
terraform.tfstate" on every redeploy after the first (databricks/cli#4986), which bricks
the bundle until you switch to DATABRICKS_BUNDLE_ENGINE=direct. D6 — the schema
GRANT-ordering bug (databricks/cli#4573): the first deploy to a fresh workspace fails
with "User does not have CREATE TABLE on Schema", but the second succeeds because the
first applied the grants before dying.
Deploy-time infrastructure. D8 — rotating a workspace's customer-managed key (CMK) requires terminating every cluster, instance pool, and SQL warehouse first: a hard maintenance window. D9 — the PrivateLink trap: enabling PrivateLink for the control plane is mistaken for "all traffic is private," but the data plane's S3 / STS / Kinesis calls still traverse the NAT, billing NAT-processing + cross-AZ transfer until each gets its own VPC endpoint.
The two hooks (this is the pack's only two-hook skill). A PreToolUse hook
(hooks/bundle-deploy-guard.py) runs before every databricks bundle deploy: it
validates the bundle's local terraform state parses as JSON, caches a timestamped
known-good backup as a recovery escape hatch, and warns loudly if the state is corrupt
or has shrunk (the D5 signature). A PostToolUse hook (hooks/bundle-grant-retry.py)
watches a deploy's output and — ONLY on the exact D6 "does not have … on Schema"
signature — adds context recommending one retry, with the reason. Both are advisory:
the pre-hook never blocks a deploy, and the post-hook never masks a real error — it
surfaces the diagnosis and stops recommending retries if the same failure persists.
Deterministic work lives in scripts/; deep knowledge in references/. The
import-uc-resource-to-bundle.py D4 workaround is self-deprecating — it exists only
until #4842 closes and says so. Control-plane state comes from the
databricks-workspace-mcp server (external_locations_list, storage_credentials_list)
or the CLI; advisory-mode fallback accepts pasted input.
Prerequisites
- Databricks CLI authenticated, plus
terraformandjqon PATH — for the bundle, import, and state operations. databricks-workspace-mcpregistered (optional) — forexternal_locations_listandstorage_credentials_list. Absent → the CLI (databricks external-locations list) or advisory mode on pasted input.- Account-level OAuth M2M (service principal) for CMK rotation — the D8 operations are account-scoped; a workspace PAT is insufficient. AWS/Azure/GCP CLI for the cloud-side key + VPC-endpoint work.
- The two hooks are plugin-level — installed with the pack. The pre-hook is silent unless a state looks corrupt; the post-hook is silent unless the exact D6 error fires.
Instructions
Pick the flow by symptom. Always name the exact error string / issue id —
unexpected EOF reading terraform.tfstate (#4986), does not have CREATE TABLE on Schema (#4573), the bundle bind UC gap (#4842) — an operator greps for those.
Step 1: A bundle deploy that fails on state (D5)
"unexpected EOF reading terraform.tfstate" after the first deploy is #4986. The
bundle-deploy-guard PreToolUse hook already cached a known-good backup; restore it,
or switch engines:
DATABRICKS_BUNDLE_ENGINE=direct databricks bundle deploy -t "$TARGET"
Engine tradeoffs + the migration steps:
${CLAUDE_SKILL_DIR}/references/bundle-engine-tradeoffs.md.
Step 2: First deploy fails "does not have CREATE TABLE on Schema" (D6)
This is the transient GRANT-ordering bug (#4573). The bundle-grant-retry PostToolUse
hook flags it; re-run the SAME deploy exactly once — the failed first pass already
applied the grants. If the identical error persists after one retry, it is NOT this
transient — treat it as a real missing privilege and stop retrying.
Step 3: A UC resource that will not bind (D4)
databricks bundle bind cannot take a UC catalog or external location yet (#4842).
Generate a review-first Terraform import plan (never hand-edit state):
python3 "${CLAUDE_SKILL_DIR}/scripts/import-uc-resource-to-bundle.py" \
--type external_location --name raw_zone --resource-key raw_zone_loc
The three workarounds, ranked by risk:
${CLAUDE_SKILL_DIR}/references/uc-resource-binding-workarounds.md.
Step 4: Rotate a customer-managed key (D8)
CMK rotation needs the whole workspace drained. Inventory the running compute (via
external_locations_list / the CLI), then plan the drain — dry-run by default:
databricks clusters list --output json > /tmp/inv-clusters.json # + warehouses, pools
python3 "${CLAUDE_SKILL_DIR}/scripts/drain-workspace.py" --inventory inv.json # dry-run
python3 "${CLAUDE_SKILL_DIR}/scripts/drain-workspace.py" --inventory inv.json --execute --manifest drain.json
# ... rotate the CMK per cloud, then:
python3 "${CLAUDE_SKILL_DIR}/scripts/drain-workspace.py" --resume drain.json
Per-cloud playbooks (AWS/Azure/GCP):
${CLAUDE_SKILL_DIR}/references/cmk-rotation-by-cloud.md.
Step 5: Audit the PrivateLink cost leak (D9)
PrivateLink covers the control plane only. Audit the data-plane VPC for the S3/STS/Kinesis endpoints and emit remediation Terraform for any that are missing:
python3 "${CLAUDE_SKILL_DIR}/scripts/audit-vpc-endpoints.py" \
--present s3 --vpc-id vpc-abc --region us-east-1 --emit-terraform
The full per-service leak/fix map:
${CLAUDE_SKILL_DIR}/references/cost-leak-map.md.
Output
- A hook backup + advisory — before each deploy, a validated known-good state backup and, if the state is corrupt/shrunk, the #4986 recovery message.
- A retry recommendation — on the exact D6 signature, a one-retry recommendation with the reason (never a masked error).
- A UC import plan — the
resources:YAML + Terraformimportblock/command to adopt an existing UC resource, with the self-deprecation notice. - A drain / resume plan — the idempotent list of compute to terminate for CMK rotation and the manifest to resume exactly what was drained.
- A VPC-endpoint verdict — SAFE or a COST LEAK finding naming the missing S3/STS/Kinesis endpoints, with remediation Terraform.
Error Handling
| Error | Cause | Solution |
|---|---|---|
unexpected EOF reading terraform.tfstate |
Terraform-engine state read bug on redeploy (D5, #4986) | Restore the guard's cached backup, or DATABRICKS_BUNDLE_ENGINE=direct. |
User does not have CREATE TABLE on Schema '…' |
DAB GRANT-ordering (D6, #4573) | Re-run the deploy once; grants were applied by the failed pass. Persists after one retry → real missing grant. |
bundle bind rejects a UC catalog / external location |
Unsupported in the CLI (D4, #4842) | Use import-uc-resource-to-bundle.py to generate a Terraform import plan; never hand-edit state. |
| CMK rotation rejected — compute still running | CMK update requires a drained workspace (D8) | drain-workspace.py --execute, rotate, then --resume. |
| High NAT / cross-AZ cost after PrivateLink | S3/STS/Kinesis have no VPC endpoint (D9) | audit-vpc-endpoints.py --emit-terraform; add Gateway (S3) + Interface (STS/Kinesis) endpoints. |
Examples
Example 1: "My second bundle deploy fails with unexpected EOF reading terraform.tfstate."
D5 (#4986). The bundle-deploy-guard hook already cached a known-good state before the
deploy — restore it, then redeploy with DATABRICKS_BUNDLE_ENGINE=direct (the direct
engine never reads the state file, so the EOF class cannot fire).
Example 2: "First deploy to a fresh workspace: User does not have CREATE TABLE on Schema."
D6 (#4573). The bundle-grant-retry hook recognises the exact signature and recommends
one retry — the failed first deploy applied the schema grants, so the second succeeds.
Example 3: "I need to bring an existing external location under my bundle."
D4 (#4842) — bundle bind can't. import-uc-resource-to-bundle.py --type external_location emits the resources: YAML plus a Terraform import block to adopt
it without recreating it or hand-editing state. The script self-deprecates when #4842 closes.
Example 4: "We turned on PrivateLink but the NAT bill went up."
D9. audit-vpc-endpoints.py --present s3 finds STS and Kinesis have no Interface
endpoint, so credential-vending and log traffic still cross the NAT — it emits the
remediation Terraform for both.
Resources
${CLAUDE_SKILL_DIR}/references/uc-resource-binding-workarounds.md— the D4bundle bindgap + 3 ranked workarounds (#4842).${CLAUDE_SKILL_DIR}/references/bundle-engine-tradeoffs.md— Terraform vs direct engine, the D5 EOF bug (#4986).${CLAUDE_SKILL_DIR}/references/cmk-rotation-by-cloud.md— AWS/Azure/GCP CMK rotation playbooks + the drain requirement (D8).${CLAUDE_SKILL_DIR}/references/cost-leak-map.md— the PrivateLink S3/STS/Kinesis NAT cost leak + fix map (D9).${CLAUDE_SKILL_DIR}/scripts/import-uc-resource-to-bundle.py— self-deprecating UC-resource import-plan generator (D4).${CLAUDE_SKILL_DIR}/scripts/drain-workspace.py— idempotent drain + resume for CMK rotation (D8).${CLAUDE_SKILL_DIR}/scripts/audit-vpc-endpoints.py— VPC-endpoint audit + remediation Terraform (D9).${CLAUDE_SKILL_DIR}/hooks/bundle-deploy-guard.py— PreToolUse state backup + validation (D5).${CLAUDE_SKILL_DIR}/hooks/bundle-grant-retry.py— PostToolUse D6 retry recommendation (D6).- Databricks Asset Bundles · Customer-managed keys · PrivateLink