databricks-solutions Databricks Workspace Configuration
POST-DEPLOYMENT VERIFICATION IS MANDATORY. Whenever you finish standing up a new workspace, a new cluster policy, a new SQL warehouse, or a new job-compute pattern, you MUST run all three paths from
deployment-verification/SKILL.md(classic cluster + serverless SQL warehouse + serverless notebook job, all against a UC table). One-path verification is incomplete work. Read that skill before you call anything "done."
How to interact with the customer
Apply MINIMAL pushback. These are day-2 operations -- the customer generally knows what they want.
- PAT with no expiry requested -- suggest once: "Want to set a 90-day lifetime? Tokens without expiry are a security risk if leaked." If they say no, create it without expiry.
- Disabling IP access lists -- confirm once: "This removes network restrictions -- all IPs will be able to reach the workspace API. Proceed?" Then do it.
- terraform destroy -- ALWAYS confirm before executing. This is irreversible. List what will be destroyed and get explicit yes/no.
- Everything else -- just execute. No pushback on warehouse sizes, policy definitions, secret values, or config keys.
SQL warehouses
Manage serverless and pro SQL warehouses for BI and ad-hoc queries.
- List existing warehouses to see names, sizes, states, and types.
- Create with name, cluster_size (2X-Small through 4X-Large), auto_stop_mins (default 15), and warehouse_type.
- Start a stopped warehouse by ID.
- Stop a running warehouse by ID.
Recommend PRO warehouse type for any workspace with Unity Catalog. PRO supports fine-grained access control, data lineage, and serverless scaling. CLASSIC warehouses lack UC-aware features.
Cluster policies
Control what users can configure when creating clusters.
- List all policies in the workspace.
- Create a policy with a name and JSON definition that restricts spark_version, node_type_id, autoscale ranges, and other cluster attributes.
- Get a specific policy by ID to inspect its definition.
- Delete a policy by ID.
Example restrictions: fix Spark version to a specific LTS release, allowlist specific node types, cap max_workers to prevent runaway costs.
Secret scopes
Store sensitive values (API keys, connection strings, passwords) that notebooks and jobs can reference without exposing plaintext.
- Create scope with a name. Scopes are workspace-level.
- Put secret into a scope with a key and string value.
- List secrets in a scope. Returns metadata only (keys and timestamps) -- secret values are never returned by the API.
- Delete a secret or scope.
IP access lists
Restrict which IP addresses can reach the workspace API and UI.
- Create an allow list (only these IPs can connect) or deny list (block these IPs).
- List existing access lists to audit current restrictions.
IP access lists must be enabled via workspace settings before they take effect. Creating a list does not automatically enable enforcement.
CRITICAL: Self-lockout prevention. When creating an IP access list, ALWAYS include the deployer's current IP address. If the allow list is enabled without the deployer's IP, Terraform loses API access to the workspace and cannot fix the list — manual intervention via the Azure/AWS console is required.
Pattern — auto-detect and include deployer IP:
data "http" "deployer_ip" {
url = "https://ifconfig.me"
}
resource "databricks_ip_access_list" "allow_list" {
label = "allow_in"
list_type = "ALLOW"
ip_addresses = concat(
var.allowed_ips,
["${chomp(data.http.deployer_ip.response_body)}/32"]
)
}
Recovery if locked out: disable IP access lists via the cloud console (Azure portal > Databricks workspace > Networking), fix the Terraform config, then re-apply.
Token management
Manage personal access tokens (PATs) for API authentication.
- List existing tokens to see comments, creation dates, and expiry.
- Create a token with a comment describing its purpose and lifetime_seconds for expiry. Best practice: always set lifetime_seconds. A 90-day lifetime (7776000 seconds) is a reasonable default.
- Revoke a token by ID to immediately invalidate it.
Workspace settings
Read and update workspace-level configuration keys.
Common settings:
enableTokensConfig-- enable/disable PAT creation for the workspace.maxTokenLifetimeDays-- enforce a maximum token lifetime (0 means no limit).enableIpAccessLists-- enable/disable IP access list enforcement.
Get a setting to see its current value. Set a setting to change it. Some settings require workspace restart to take effect.
Destroying infrastructure
Tear down Terraform-managed infrastructure.
- List past Terraform runs to find the deployment you want to destroy.
- Get outputs from a run to see what resources exist (workspace URLs, resource IDs).
- Destroy a run to tear down all resources it created.
ALWAYS ask for user confirmation before destroying. Show the user what will be destroyed (workspace name, resource group, VPC, etc.) and wait for explicit approval. Terraform destroy is irreversible -- deleted workspaces, storage accounts, and networking resources cannot be recovered.
Workspace name reuse after destroy
When you destroy a Databricks workspace, the workspace name enters a BANNED state on the account side for approximately 1 hour before it can be reused. Symptom on immediate re-create:
Error: workspace name "<name>" is currently BANNED — try again later or pick a new name
- GCP and AWS: consistent ~1hr delay
- Azure: the workspace resource name is managed by Azure, but the Databricks-side display name follows the same rule
Two mitigation patterns for stress-test / dev-loop scenarios where destroy/recreate is common:
- Append a short random suffix to the workspace name:
${var.prefix}-ws-${random_id.suffix.hex} - Use a date-suffixed name:
${var.prefix}-ws-$(date +%Y%m%d-%H%M)
For real production destroys, the 1-hour cool-down is rarely a blocker. Just communicate it to the customer if they want immediate recreation under the same name.
Cross-links
- For workspace creation and infrastructure provisioning, see platform-provisioning.
- For permissions, groups, and access control, see identity-governance.