Databricks Lakebase Autoscale

Lakebase Autoscaling

Patterns and best practices for using Lakebase Autoscaling, the next-generation managed PostgreSQL on Databricks with autoscaling compute, branching, scale-to-zero, and instant restore.

When to Use

Use this skill when:

• Building applications that need a PostgreSQL database with autoscaling compute
• Working with database branching for dev/test/staging workflows
• Adding persistent state to applications with scale-to-zero cost savings
• Implementing reverse ETL from Delta Lake to an operational database via synced tables
• Managing Lakebase Autoscaling projects, branches, computes, or credentials

Overview

Lakebase Autoscaling is Databricks' next-generation managed PostgreSQL service for OLTP workloads. It provides autoscaling compute, Git-like branching, scale-to-zero, and instant point-in-time restore.

Available Regions (AWS): us-east-1, us-east-2, eu-central-1, eu-west-1, eu-west-2, ap-south-1, ap-southeast-1, ap-southeast-2

Available Regions (Azure Beta): eastus2, westeurope, westus

Project Hierarchy

Understanding the hierarchy is essential for working with Lakebase Autoscaling:

Project (top-level container)
  └── Branch(es) (isolated database environments)
        ├── Compute (primary R/W endpoint)
        ├── Read Replica(s) (optional, read-only)
        ├── Role(s) (Postgres roles)
        └── Database(s) (Postgres databases)
              └── Schema(s)

Quick Start

Create a project and connect:

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.postgres import Project, ProjectSpec

w = WorkspaceClient()

# Create a project (long-running operation)
operation = w.postgres.create_project(
    project=Project(
        spec=ProjectSpec(
            display_name="My Application",
            pg_version="17"
        )
    ),
    project_id="my-app"
)
result = operation.wait()
print(f"Created project: {result.name}")

Common Patterns

Generate OAuth Token

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

# Generate database credential for connecting (optionally scoped to an endpoint)
cred = w.postgres.generate_database_credential(
    endpoint="projects/my-app/branches/production/endpoints/ep-primary"
)
token = cred.token  # Use as password in connection string
# Token expires after 1 hour

Connect from Notebook

import psycopg
from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

# Get endpoint details
endpoint = w.postgres.get_endpoint(
    name="projects/my-app/branches/production/endpoints/ep-primary"
)
host = endpoint.status.hosts.host

# Generate token (scoped to endpoint)
cred = w.postgres.generate_database_credential(
    endpoint="projects/my-app/branches/production/endpoints/ep-primary"
)

# Connect using psycopg3
conn_string = (
    f"host={host} "
    f"dbname=databricks_postgres "
    f"user={w.current_user.me().user_name} "
    f"password={cred.token} "
    f"sslmode=require"
)
with psycopg.connect(conn_string) as conn:
    with conn.cursor() as cur:
        cur.execute("SELECT version()")
        print(cur.fetchone())

Create a Branch for Development

from databricks.sdk.service.postgres import Branch, BranchSpec, Duration

# Create a dev branch with 7-day expiration
branch = w.postgres.create_branch(
    parent="projects/my-app",
    branch=Branch(
        spec=BranchSpec(
            source_branch="projects/my-app/branches/production",
            ttl=Duration(seconds=604800)  # 7 days
        )
    ),
    branch_id="development"
).wait()
print(f"Branch created: {branch.name}")

Resize Compute (Autoscaling)

from databricks.sdk.service.postgres import Endpoint, EndpointSpec, FieldMask

# Update compute to autoscale between 2-8 CU
w.postgres.update_endpoint(
    name="projects/my-app/branches/production/endpoints/ep-primary",
    endpoint=Endpoint(
        name="projects/my-app/branches/production/endpoints/ep-primary",
        spec=EndpointSpec(
            autoscaling_limit_min_cu=2.0,
            autoscaling_limit_max_cu=8.0
        )
    ),
    update_mask=FieldMask(field_mask=[
        "spec.autoscaling_limit_min_cu",
        "spec.autoscaling_limit_max_cu"
    ])
).wait()

MCP Tools

The following MCP tools are available for managing Lakebase infrastructure. Use type="autoscale" for Lakebase Autoscaling.

Database (Project) Management

Branch Management

Credentials

Reference Files

• projects.md - Project management patterns and settings
• branches.md - Branching workflows, protection, and expiration
• computes.md - Compute sizing, autoscaling, and scale-to-zero
• connection-patterns.md - Connection patterns for different use cases
• reverse-etl.md - Synced tables from Delta Lake to Lakebase

CLI Quick Reference

# Create a project
databricks postgres create-project \
    --project-id my-app \
    --json '{"spec": {"display_name": "My App", "pg_version": "17"}}'

# List projects
databricks postgres list-projects

# Get project details
databricks postgres get-project projects/my-app

# Create a branch
databricks postgres create-branch projects/my-app development \
    --json '{"spec": {"source_branch": "projects/my-app/branches/production", "no_expiry": true}}'

# List branches
databricks postgres list-branches projects/my-app

# Get endpoint details
databricks postgres get-endpoint projects/my-app/branches/production/endpoints/ep-primary

# Delete a project
databricks postgres delete-project projects/my-app

Key Differences from Lakebase Provisioned

Common Issues

Current Limitations

These features are NOT yet supported in Lakebase Autoscaling:

• High availability with readable secondaries (use read replicas instead)
• Databricks Apps UI integration (Apps can connect manually via credentials)
• Feature Store integration
• Stateful AI agents (LangChain memory)
• Postgres-to-Delta sync (only Delta-to-Postgres reverse ETL)
• Custom billing tags and serverless budget policies
• Direct migration from Lakebase Provisioned (use pg_dump/pg_restore or reverse ETL)

SDK Version Requirements

• Databricks SDK for Python: >= 0.81.0 (for w.postgres module)
• psycopg: 3.x (supports hostaddr parameter for DNS workaround)
• SQLAlchemy: 2.x with postgresql+psycopg driver

%pip install -U "databricks-sdk>=0.81.0" "psycopg[binary]>=3.0" sqlalchemy

Notes

• Compute Units in Autoscaling provide ~2 GB RAM each (vs 16 GB in Provisioned).
• Resource naming follows hierarchical paths: projects/{id}/branches/{id}/endpoints/{id}.
• All create/update/delete operations are long-running -- use .wait() in the SDK.
• Tokens are short-lived (1 hour) -- production apps MUST implement token refresh.
• Postgres versions 16 and 17 are supported.

Related Skills

• databricks-lakebase-provisioned - fixed-capacity managed PostgreSQL (predecessor)
• databricks-app-apx - full-stack apps that can use Lakebase for persistence
• databricks-app-python - Python apps with Lakebase backend
• databricks-python-sdk - SDK used for project management and token generation
• databricks-asset-bundles - deploying apps with Lakebase resources
• databricks-jobs - scheduling reverse ETL sync jobs