433 Asset Bundles E0cde027

Databricks Asset Bundles (DABs)

Databricks Asset Bundles provide Infrastructure-as-Code for Databricks resources, enabling version control, automated deployments, and environment management.

What are Asset Bundles?

Asset Bundles let you define your Databricks projects as code, including:

• Jobs
• Pipelines (Delta Live Tables)
• Apps
• Models
• Dashboards
• Notebooks
• Python files
• Configuration files

Bundle Commands

# Initialize a new bundle from template
databricks bundle init --profile my-workspace

# Validate bundle configuration
databricks bundle validate --profile my-workspace

# Deploy bundle to workspace
databricks bundle deploy --profile my-workspace

# Deploy to specific environment (dev/staging/prod)
databricks bundle deploy -t dev --profile my-workspace
databricks bundle deploy -t staging --profile my-workspace
databricks bundle deploy -t prod --profile my-workspace

# Run a resource from the bundle
databricks bundle run <resource-name> --profile my-workspace

# Generate configuration for existing resources
databricks bundle generate job <job-id> --profile my-workspace
databricks bundle generate pipeline <pipeline-id> --profile my-workspace
databricks bundle generate dashboard <dashboard-id> --profile my-workspace
databricks bundle generate app <app-name> --profile my-workspace

# Destroy bundle resources (use with caution!)
databricks bundle destroy --profile my-workspace
databricks bundle destroy -t dev --profile my-workspace

Bundle Structure

A typical bundle has this structure:

my-project/
├── databricks.yml         # Main bundle configuration
├── resources/
│   ├── jobs/
│   │   └── my_job.yml
│   ├── pipelines/
│   │   └── my_pipeline.yml
│   ├── apps/
│   │   └── my_app.yml
│   └── models/
│       └── my_model.yml
├── src/
│   ├── notebooks/
│   │   └── process_data.py
│   └── python/
│       └── utils.py
├── tests/
│   └── test_utils.py
└── README.md

Main Configuration (databricks.yml)

Basic Example

bundle:
  name: my-project

# Workspace configuration
workspace:
  host: https://company-workspace.cloud.databricks.com

# Define targets (environments)
targets:
  dev:
    default: true
    mode: development
    workspace:
      root_path: /Users/${workspace.current_user.userName}/.bundle/${bundle.name}/dev

  staging:
    mode: production
    workspace:
      root_path: /Shared/.bundle/${bundle.name}/staging

  prod:
    mode: production
    workspace:
      root_path: /Shared/.bundle/${bundle.name}/prod

# Include resource definitions
include:
  - resources/**/*.yml

Advanced Example with Variables

bundle:
  name: my-project

variables:
  catalog:
    description: "Unity Catalog name"
    default: dev_catalog

  warehouse_id:
    description: "SQL Warehouse ID"

workspace:
  host: https://company-workspace.cloud.databricks.com

targets:
  dev:
    default: true
    mode: development
    variables:
      catalog: dev_catalog
      warehouse_id: "abc123def456"
    workspace:
      root_path: /Users/${workspace.current_user.userName}/.bundle/${bundle.name}/dev

  prod:
    mode: production
    variables:
      catalog: prod_catalog
      warehouse_id: "xyz789uvw012"
    workspace:
      root_path: /Shared/.bundle/${bundle.name}/prod

include:
  - resources/**/*.yml

Initializing a Bundle

Using Templates

# Start initialization (interactive)
databricks bundle init --profile my-workspace

Available templates:

• default-python - Python project with jobs
• dlt-pipeline - Delta Live Tables pipeline
• ml-project - ML project with MLflow
• databricks-app - Databricks App

Template Selection Example

$ databricks bundle init

Template to use [default-python, dlt-pipeline, ml-project, databricks-app]: default-python
Project name: my-analytics-project

Defining Resources

Job Resource

# resources/jobs/daily_job.yml
resources:
  jobs:
    daily_analytics:
      name: "Daily Analytics Job"

      tasks:
        - task_key: "extract_data"
          notebook_task:
            notebook_path: "./src/notebooks/extract.py"
            source: WORKSPACE
          new_cluster:
            spark_version: "13.3.x-scala2.12"
            node_type_id: "i3.xlarge"
            num_workers: 2

        - task_key: "transform_data"
          depends_on:
            - task_key: "extract_data"
          notebook_task:
            notebook_path: "./src/notebooks/transform.py"
            source: WORKSPACE
          new_cluster:
            spark_version: "13.3.x-scala2.12"
            node_type_id: "i3.xlarge"
            num_workers: 2

      schedule:
        quartz_cron_expression: "0 0 2 * * ?"  # Daily at 2 AM
        timezone_id: "UTC"

Pipeline Resource (DLT)

# resources/pipelines/data_pipeline.yml
resources:
  pipelines:
    customer_pipeline:
      name: "Customer Data Pipeline"

      catalog: ${var.catalog}
      target: customer_analytics

      libraries:
        - notebook:
            path: ./src/notebooks/customer_pipeline.py

      clusters:
        - label: default
          autoscale:
            min_workers: 1
            max_workers: 5
            mode: ENHANCED

App Resource

# resources/apps/my_app.yml
resources:
  apps:
    dashboard_app:
      name: "analytics-dashboard"
      description: "Customer analytics dashboard"
      source_code_path: ./src/app

Model Resource

# resources/models/my_model.yml
resources:
  registered_models:
    customer_churn:
      name: "${var.catalog}.${var.schema}.customer_churn_model"
      description: "Customer churn prediction model"

Working with Environments

Environment Configuration

Environments (targets) allow you to deploy the same code to different workspaces with different configurations.

targets:
  dev:
    default: true
    mode: development
    variables:
      catalog: dev_catalog
      cluster_size: "Small"
    workspace:
      root_path: /Users/${workspace.current_user.userName}/.bundle/${bundle.name}/dev

  staging:
    mode: production
    variables:
      catalog: staging_catalog
      cluster_size: "Medium"
    workspace:
      root_path: /Shared/.bundle/${bundle.name}/staging
      host: https://staging-workspace.cloud.databricks.com

  prod:
    mode: production
    variables:
      catalog: prod_catalog
      cluster_size: "Large"
    workspace:
      root_path: /Shared/.bundle/${bundle.name}/prod
      host: https://prod-workspace.cloud.databricks.com

Deploying to Different Environments

# Deploy to dev (default)
databricks bundle deploy --profile my-workspace

# Deploy to staging
databricks bundle deploy -t staging --profile my-workspace

# Deploy to production
databricks bundle deploy -t prod --profile my-workspace

Bundle Workflow

Complete Development Workflow

1. Initialize bundle:

   databricks bundle init --profile my-workspace
   

2. Develop locally:

   • Edit databricks.yml and resource files
   • Write notebooks, Python scripts, SQL queries
   • Configure jobs, pipelines, apps

3. Validate configuration:

   databricks bundle validate --profile my-workspace
   

4. Deploy to development:

   databricks bundle deploy -t dev --profile my-workspace
   

5. Test your deployment:

   # Run a job
   databricks bundle run my-job -t dev --profile my-workspace
   
   # Start a pipeline
   databricks bundle run my-pipeline -t dev --profile my-workspace
   

6. Deploy to production:

   databricks bundle deploy -t prod --profile my-workspace
   

Generating Bundle from Existing Resources

If you have existing resources in your workspace, you can generate bundle configuration:

Generate Job Configuration

# Get job ID from list
databricks jobs list --profile my-workspace

# Generate configuration
databricks bundle generate job 12345 --profile my-workspace

This creates:

• Job configuration file in resources/
• Downloads associated notebooks/files

Generate Pipeline Configuration

databricks bundle generate pipeline <pipeline-id> --profile my-workspace

Generate App Configuration

databricks bundle generate app my-app --profile my-workspace

This generates:

• App configuration file
• Downloads app source code

Generate Dashboard Configuration

databricks bundle generate dashboard <dashboard-id> --profile my-workspace

Binding Existing Resources

You can link bundle-defined resources to existing resources in your workspace without recreating them:

databricks bundle bind --profile my-workspace

This is useful when:

• Migrating existing projects to bundles
• Adopting bundle management for running resources
• Avoiding downtime during migration

Variables and Templating

Defining Variables

# databricks.yml
variables:
  catalog:
    description: "Unity Catalog name"
    default: dev_catalog

  warehouse_id:
    description: "SQL Warehouse ID"

  num_workers:
    description: "Number of cluster workers"
    default: 2

Using Variables

# In resource files
resources:
  jobs:
    my_job:
      name: "Job in ${var.catalog}"

      tasks:
        - task_key: "main"
          notebook_task:
            notebook_path: "./notebook.py"
          new_cluster:
            num_workers: ${var.num_workers}

Environment-Specific Variables

targets:
  dev:
    variables:
      catalog: dev_catalog
      num_workers: 2

  prod:
    variables:
      catalog: prod_catalog
      num_workers: 10

Best Practices

1. Use Version Control

Always commit your bundle to Git:

git init
git add databricks.yml resources/ src/
git commit -m "Initial bundle setup"

2. Separate Concerns

Organize resources by type:

resources/
├── jobs/
│   ├── daily_jobs.yml
│   └── hourly_jobs.yml
├── pipelines/
│   ├── ingestion.yml
│   └── transformation.yml
└── apps/
    └── dashboard.yml

3. Use Environment-Specific Configuration

targets:
  dev:
    mode: development  # Allows in-place updates

  prod:
    mode: production   # Ensures immutable deployments

4. Validate Before Deploy

Always validate:

databricks bundle validate --profile my-workspace

5. Use Descriptive Names

resources:
  jobs:
    daily_customer_analytics:  # Good: descriptive
      name: "Daily Customer Analytics"

6. Document with Comments

# This job runs daily ETL for customer analytics
# Schedule: 2 AM UTC daily
# Dependencies: Unity Catalog tables in analytics schema
resources:
  jobs:
    daily_etl:
      name: "Daily ETL"
      # ...

7. Use Variables for Environment Differences

variables:
  catalog: ${bundle.environment}  # Automatically uses env name
  cluster_size: Small

targets:
  prod:
    variables:
      cluster_size: Large  # Override for prod

Troubleshooting

Bundle Validation Errors

Symptom: databricks bundle validate shows errors

Solution:

1. Check YAML syntax (proper indentation, no tabs)
2. Verify all required fields are present
3. Check that resource references are correct
4. Use databricks bundle validate --debug for detailed errors

Deployment Fails

Symptom: databricks bundle deploy fails

Solution:

1. Run validation first: databricks bundle validate
2. Check workspace permissions
3. Verify target configuration
4. Check for resource name conflicts
5. Review error message for specific issues

Resources Not Found After Deploy

Symptom: Resources deployed but not visible in workspace

Solution:

1. Check target root_path in databricks.yml
2. Verify you're looking in the correct workspace location
3. Check deployment logs for errors
4. Ensure resources were included in deployment

Variable Not Resolved

Symptom: Variable showing as ${var.name} instead of actual value

Solution:

1. Check variable is defined in databricks.yml
2. Verify variable has value in target
3. Use correct syntax: ${var.variable_name}
4. Check variable scope (bundle vs target)

Related Topics

• Data Exploration - Validate data exposed by bundle deployments
• Apps - Define app resources (use databricks-apps skill for full app development)