Databricks Unity Catalog

Databricks Unity Catalog Management

Discover, create, and manage Unity Catalog resources (catalogs, schemas, tables) via MCP tools. Enforces best practices for naming, governance, and metadata.

When to Use This Skill

• Exploring available catalogs and schemas
• Creating schemas for new projects
• Discovering tables and their metadata
• Setting up data governance (owners, comments)
• Managing table lifecycle (create, update, delete)
• Validating catalog structure before pipeline deployment

Core Concepts

Unity Catalog Hierarchy

Catalog (highest level)
  └── Schema (database)
      └── Table (data asset)
          ├── Columns
          ├── Properties
          └── Permissions

Resource Naming

Three-level namespace: catalog.schema.table

Example: ml_production.feature_store.customer_features

Discovery Workflows

Workflow 1: Explore Catalog Structure

Discover what's available in Unity Catalog.

Pattern:

1. Use list_catalogs MCP tool → Get all catalogs
2. Use get_catalog → Get details for specific catalog
3. Use list_schemas → Get schemas in catalog
4. Use list_tables → Get tables in schema
5. Use get_table → Get table details including columns

Example:

# User: "What tables are available in my catalog?"

# Step 1: List all catalogs
list_catalogs()
# Returns: [{name: "main", name: "ml_dev", name: "ml_prod", ...}]

# Step 2: List schemas in a catalog
list_schemas(catalog_name="ml_dev")
# Returns: [{name: "bronze", name: "silver", name: "gold", ...}]

# Step 3: List tables in a schema
list_tables(catalog_name="ml_dev", schema_name="silver")
# Returns: [{name: "clean_events", name: "customer_features", ...}]

# Step 4: Get table details
get_table(full_table_name="ml_dev.silver.clean_events")
# Returns: {
#   name: "clean_events",
#   columns: [{name: "event_id", type: "string"}, ...],
#   table_type: "MANAGED",
#   owner: "user@company.com",
#   comment: "Cleaned event data from bronze layer"
# }

Workflow 2: Validate Table Schema

Check table structure before using in pipeline.

Pattern:

1. Use get_table with full table name
2. Examine columns array
3. Check data types
4. Verify required columns exist

Example:

# User: "Check if the customer table has email column"

get_table(full_table_name="ml_prod.gold.customers")
# Returns table metadata including:
# columns: [
#   {name: "customer_id", type: "bigint"},
#   {name: "email", type: "string"},
#   {name: "created_date", type: "date"}
# ]

# Verify: Yes, email column exists with type string

Schema Management

Workflow 3: Create Schema for New Project

Set up schema with proper naming and metadata.

Pattern:

1. Verify catalog exists with get_catalog
2. Create schema with create_schema:
   • Use lowercase with underscores
   • Add descriptive comment
   • System sets owner automatically
3. Verify creation with get_schema

Example:

# User: "Create a schema for my ML feature engineering project"

# Step 1: Verify catalog exists
get_catalog(catalog_name="ml_dev")
# Returns: {name: "ml_dev", owner: "...", ...}

# Step 2: Create schema
create_schema(
    catalog_name="ml_dev",
    schema_name="feature_engineering",
    comment="Feature engineering workspace for customer churn prediction model. Contains raw features, engineered features, and training datasets."
)
# Returns: Schema created successfully

# Step 3: Verify
get_schema(full_schema_name="ml_dev.feature_engineering")
# Returns: {
#   name: "feature_engineering",
#   full_name: "ml_dev.feature_engineering",
#   comment: "Feature engineering workspace for...",
#   owner: "user@company.com"
# }

Workflow 4: Update Schema Metadata

Modify schema properties after creation.

Pattern:

1. Use update_schema with full schema name
2. Provide new values for:
   • comment - Updated description
   • owner - New owner email
   • new_name - Rename schema (optional)

Example:

# User: "Update the schema description"

update_schema(
    full_schema_name="ml_dev.feature_engineering",
    comment="UPDATED: Feature engineering workspace for customer churn prediction v2. Includes time-series features and demographic aggregations.",
    owner="ml-team@company.com"
)
# Returns: Schema updated successfully

Table Management

Workflow 5: Get Table Information

Retrieve comprehensive table metadata.

Pattern:

1. Use get_table with full three-level name
2. Examine returned metadata:
   • Columns and types
   • Table type (MANAGED vs EXTERNAL)
   • Owner and permissions
   • Properties and comments
   • Storage location

Example:

# User: "Show me the structure of the features table"

get_table(full_table_name="ml_prod.features.customer_360")
# Returns: {
#   name: "customer_360",
#   catalog_name: "ml_prod",
#   schema_name: "features",
#   table_type: "MANAGED",
#   columns: [
#     {name: "customer_id", type_name: "BIGINT", comment: "Primary key"},
#     {name: "lifetime_value", type_name: "DOUBLE", comment: "Total spend"},
#     {name: "churn_score", type_name: "DOUBLE", comment: "0-1 churn probability"},
#     {name: "last_updated", type_name: "TIMESTAMP", comment: "Feature refresh timestamp"}
#   ],
#   owner: "ml-team@company.com",
#   comment: "360-degree customer features for ML models"
# }

Workflow 6: Delete Table

Remove table from Unity Catalog.

Pattern:

1. Confirm with user before deleting
2. Use delete_table with full table name
3. For MANAGED tables: data is deleted too
4. For EXTERNAL tables: only metadata removed

Example:

# User: "Delete the old test table"

# IMPORTANT: Always confirm deletions
# Ask user: "Are you sure you want to delete ml_dev.test.old_data? This cannot be undone."

# If confirmed:
delete_table(full_table_name="ml_dev.test.old_data")
# Returns: Table deleted successfully

Workflow 7: Delete Schema

Remove schema and all its tables.

Pattern:

1. Confirm with user (DANGEROUS operation)
2. List tables in schema first to show what will be deleted
3. Use delete_schema with full schema name
4. All tables in schema are deleted too

Example:

# User: "Delete the temp schema"

# IMPORTANT: List contents first
list_tables(catalog_name="ml_dev", schema_name="temp")
# Shows: 3 tables will be deleted

# Ask user: "Are you sure? This will delete the schema 'ml_dev.temp' and all 3 tables. This cannot be undone."

# If confirmed:
delete_schema(full_schema_name="ml_dev.temp")
# Returns: Schema deleted successfully

Naming Conventions

Required Naming Rules

DO:

• Use lowercase letters only
• Use underscores to separate words
• Use descriptive names
• Examples:
  • ml_production
  • feature_store
  • customer_features
  • daily_aggregates

DON'T:

• Use hyphens: ml-production ❌
• Use spaces: ml production ❌
• Use special characters: ml_prod! ❌
• Use camelCase: mlProduction ❌
• Use numbers at start: 2024_features ❌

Recommended Naming Patterns

Catalogs:

• <domain>_<environment>: ml_dev, ml_prod, analytics_staging
• <team>_<purpose>: data_eng_pipelines, ml_models

Schemas:

• Medallion layers: bronze, silver, gold
• By function: feature_store, models, metrics
• By project: churn_prediction, recommendation_engine

Tables:

• Include domain: customer_features, order_metrics
• Include grain: daily_sales, hourly_events
• Include status: active_users, archived_orders

Best Practices

1. Always Add Comments

Every schema and table should have a descriptive comment explaining:

• What data it contains
• Who owns/maintains it
• When/how it's updated
• Related upstream/downstream dependencies

# Good
create_schema(
    catalog_name="ml_dev",
    schema_name="feature_engineering",
    comment="Feature engineering workspace for churn model v2. Updated daily at 2am UTC. Owned by ML team."
)

# Bad - no comment
create_schema(
    catalog_name="ml_dev",
    schema_name="feature_engineering"
)

2. Use MANAGED Tables by Default

Unless you have external data in cloud storage:

• Use MANAGED tables (Databricks manages storage)
• Data deleted when table dropped
• Simpler governance

Use EXTERNAL only when:

• Data shared across multiple systems
• Need independent data lifecycle
• Existing data in cloud storage

3. Set Proper Owners

• Individual for personal dev: user@company.com
• Team/group for shared: ml-team@company.com
• Enables proper governance and access control

4. Verify Before Using

Always verify structure before using tables:

# Before writing pipeline
table_info = get_table(full_table_name="catalog.schema.table")

# Check required columns exist
required_columns = {"customer_id", "purchase_date", "amount"}
actual_columns = {col["name"] for col in table_info["columns"]}

if not required_columns.issubset(actual_columns):
    missing = required_columns - actual_columns
    print(f"ERROR: Missing required columns: {missing}")

5. Use Three-Level Names

Always use full names for clarity:

# Good - explicit
get_table(full_table_name="ml_prod.features.customer_360")

# Bad - ambiguous
# "features" table in which catalog/schema?

Integration with Other Skills

Called By

• databricks-ml-pipeline - Creates schemas for models and features
• databricks-data-engineering - Creates medallion architecture schemas (bronze, silver, gold)

Works With

• databricks-testing - Tests against discovered tables

Common Patterns

Pattern: Set Up Medallion Architecture

# Create bronze, silver, gold schemas for data pipeline

catalog = "de_prod"

# Bronze layer - raw data
create_schema(
    catalog_name=catalog,
    schema_name="bronze",
    comment="Raw ingested data with minimal transformation. Append-only. Retained for 90 days."
)

# Silver layer - cleaned data
create_schema(
    catalog_name=catalog,
    schema_name="silver",
    comment="Cleaned and validated data. Deduplicated, type-cast, quality-checked. Retained for 1 year."
)

# Gold layer - business aggregates
create_schema(
    catalog_name=catalog,
    schema_name="gold",
    comment="Business-ready aggregates and metrics. Optimized for analytics queries. Retained for 3 years."
)

Pattern: Audit Schema Contents

# User: "Show me all tables in the feature store"

# Get schema details
schema_info = get_schema(full_schema_name="ml_prod.feature_store")
print(f"Schema: {schema_info['full_name']}")
print(f"Owner: {schema_info['owner']}")
print(f"Comment: {schema_info['comment']}")

# List all tables
tables = list_tables(catalog_name="ml_prod", schema_name="feature_store")

print(f"\nTables ({len(tables)}):")
for table in tables:
    # Get details for each
    table_info = get_table(full_table_name=f"ml_prod.feature_store.{table['name']}")
    print(f"  - {table_info['name']}")
    print(f"    Type: {table_info['table_type']}")
    print(f"    Columns: {len(table_info['columns'])}")
    print(f"    Comment: {table_info.get('comment', 'No description')}")

Troubleshooting

Issue: Catalog Not Found

Error: "Catalog 'xyz' does not exist"

Solution:

• Use list_catalogs() to see available catalogs
• Check spelling and case (must be lowercase)
• Verify you have access to catalog

Issue: Schema Already Exists

Error: "Schema 'catalog.schema' already exists"

Solution:

• Use get_schema() to check if exists
• Use update_schema() to modify instead of create
• Use different name or delete existing first

Issue: Invalid Name

Error: "Invalid name: contains invalid characters"

Solution:

• Use only lowercase letters, numbers, underscores
• No hyphens, spaces, or special characters
• Don't start with numbers

Security Reminders

• Never grant overly broad permissions
• Use least-privilege access
• Set appropriate owners for governance
• Document who has access in comments

Summary

This skill manages Unity Catalog resources:

• Discovery: Explore catalogs, schemas, tables
• Creation: Set up schemas with proper naming and governance
• Updates: Modify metadata, owners, comments
• Deletion: Remove schemas/tables (with confirmation)
• Best practices: Naming conventions, comments, MANAGED tables, proper ownership

Use this skill to set up catalog structure before building pipelines with other skills.