Unity Catalog
Guidance for Unity Catalog governance — access control, the privilege model,
external locations, securable DDL, and fine-grained access — plus system tables and
volume file operations.
Before running databricks CLI commands, confirm the CLI and the subcommand exist.
Run databricks --version — this skill assumes the unified CLI (≥ v1.0.0). Several
subcommands shown here (experimental aitools, system-schemas, external-lineage,
grants) vary by version or workspace availability; if one is missing or rejects a flag,
fall back to the SQL form or the Python SDK rather than guessing. Each reference notes its
own version floor where relevant.
When to Use This Skill
Use this skill when:
Governance & access control (start here):
- Granting or revoking access —
GRANT/REVOKE, the UC privilege model, ownership (ALTER … OWNER TO), SHOW GRANTS, "who can read/write this table?"
- Row- and column-level security — row filters, column masks, dynamic views with
current_user() / is_account_group_member()
- External locations & storage credentials —
CREATE STORAGE CREDENTIAL, CREATE EXTERNAL LOCATION, backing external tables/volumes
- Securable DDL & metadata — creating/altering catalogs, schemas, managed vs external tables, views; comments, tags, table properties, ownership
- Connection credentials & secrets: a connection's password from a secret scope (
secret()) and its rotation; where the secret lives
Observability & files:
- Working with volumes (upload, download, list files in
/Volumes/)
- Querying lineage (table dependencies, column-level lineage)
- Analyzing audit logs (who accessed what, permission changes)
- Monitoring billing and usage (DBU consumption, cost analysis)
- Tracking compute resources (cluster usage, warehouse metrics)
- Reviewing job execution (run history, success rates, failures)
- Analyzing query performance (slow queries, warehouse utilization)
- Profiling data quality (data profiling, drift detection, metric tables)
Reference Files
| Topic |
File |
Description |
| Access Control |
references/1-access-control.md |
Privilege model, securable hierarchy, GRANT/REVOKE, ownership, inheritance, SHOW GRANTS |
| External Locations |
references/2-external-locations.md |
Storage credentials (AWS/Azure/GCP), external locations, validation |
| Securables DDL |
references/3-securables-ddl.md |
CREATE/ALTER/DROP catalogs/schemas/tables/views, comments, tags, ownership |
| Fine-Grained Access |
references/4-fine-grained-access.md |
Row filters, column masks, dynamic views |
| System Tables |
references/5-system-tables.md |
Lineage, audit, billing, compute, jobs, query history |
| Volumes |
references/6-volumes.md |
Volume file operations, permissions, best practices |
| Data Profiling |
references/7-data-profiling.md |
Data profiling, drift detection, profile metrics |
| Connections & Secrets |
references/8-connections-and-secrets.md |
A connection's password and its rotation; where the secret lives |
Quick Start
Create Unity Catalog Objects (CLI)
Use --json for create commands. Positional argument order differs per command and
has changed across CLI versions, so --json is the order-independent, version-stable form
shown throughout this skill.
# Create a catalog
databricks catalogs create --json '{"name": "my_catalog"}'
# Create a schema
databricks schemas create --json '{"name": "my_schema", "catalog_name": "my_catalog"}'
# Create a managed volume
databricks volumes create --json '{
"catalog_name": "my_catalog",
"schema_name": "my_schema",
"name": "my_volume",
"volume_type": "MANAGED"
}'
# List catalogs, schemas, volumes (read commands take simple positional args)
databricks catalogs list
databricks schemas list my_catalog
databricks volumes list my_catalog.my_schema
Positional create args still work if you prefer them, but the order is not uniform
across commands — this is the per-command order (and the reason --json is recommended):
| Command |
Positional create order |
databricks catalogs create |
NAME |
databricks schemas create |
NAME CATALOG_NAME |
databricks volumes create |
CATALOG_NAME SCHEMA_NAME NAME VOLUME_TYPE |
CLI surface varies by version. If a databricks subcommand or positional signature is
missing in your install, prefer --json, the SQL form, or the Python SDK rather than
guessing flags.
Volume File Operations (CLI)
databricks fs requires the dbfs: scheme prefix even for UC Volume paths — without it the CLI treats the path as local filesystem and errors with no such directory.
# List files in a volume
databricks fs ls dbfs:/Volumes/catalog/schema/volume/path/
# Upload a directory's contents to a volume (-r copies contents, not the directory itself)
databricks fs cp -r --overwrite /tmp/data dbfs:/Volumes/catalog/schema/volume/dest
# Download a file from a volume
databricks fs cp dbfs:/Volumes/catalog/schema/volume/file.csv /tmp/file.csv
# Create a directory in a volume
databricks fs mkdirs dbfs:/Volumes/catalog/schema/volume/new_folder
Grant & Revoke Access
GRANT/REVOKE is the core governance operation. See references/1-access-control.md for the full privilege model.
-- Grant read access on a schema to a group
GRANT USE CATALOG ON CATALOG analytics TO `data_readers`;
GRANT USE SCHEMA ON SCHEMA analytics.gold TO `data_readers`;
GRANT SELECT ON SCHEMA analytics.gold TO `data_readers`;
-- Who can access this table?
SHOW GRANTS ON TABLE analytics.gold.customers;
-- Revoke
REVOKE SELECT ON SCHEMA analytics.gold FROM `data_readers`;
Enable System Tables Access
-- Grant access to system tables
GRANT USE CATALOG ON CATALOG system TO `data_engineers`;
GRANT USE SCHEMA ON SCHEMA system.access TO `data_engineers`;
GRANT SELECT ON SCHEMA system.access TO `data_engineers`;
Common Queries
-- Table lineage: What tables feed into this table?
SELECT source_table_full_name, source_column_name
FROM system.access.table_lineage
WHERE target_table_full_name = 'catalog.schema.table'
AND event_date >= current_date() - 7;
-- Audit: Recent permission changes
SELECT event_time, user_identity.email, action_name, request_params
FROM system.access.audit
WHERE action_name LIKE '%GRANT%' OR action_name LIKE '%REVOKE%'
ORDER BY event_time DESC
LIMIT 100;
-- Billing: DBU usage by workspace
SELECT workspace_id, sku_name, SUM(usage_quantity) AS total_dbus
FROM system.billing.usage
WHERE usage_date >= current_date() - 30
GROUP BY workspace_id, sku_name;
Running SQL from the CLI
databricks experimental aitools tools query is an experimental command. The
experimental namespace is not guaranteed to be stable across CLI versions and may be
absent in your install. Prefer running system-table SQL from a SQL warehouse (SQL
editor, scheduled query) or the Python SDK (w.statement_execution.execute_statement),
or a notebook. Use the experimental CLI only for quick ad-hoc checks.
Getting the IDs these examples use. WAREHOUSE_ID — run databricks warehouses list
(or copy it from a SQL warehouse's Connection details in the UI). METASTORE_ID (used in
references/5-system-tables.md) — w.metastores.current().metastore_id
via the SDK, or the Catalog UI → metastore details.
Experimental CLI form (convenience only):
databricks experimental aitools tools query --warehouse WAREHOUSE_ID "
SELECT source_table_full_name, target_table_full_name
FROM system.access.table_lineage
WHERE event_date >= current_date() - 7
"
Stable SDK fallback (works on any CLI version):
from databricks.sdk import WorkspaceClient
w = WorkspaceClient()
resp = w.statement_execution.execute_statement(
warehouse_id="WAREHOUSE_ID",
statement="""
SELECT source_table_full_name, target_table_full_name
FROM system.access.table_lineage
WHERE event_date >= current_date() - 7
LIMIT 100
""",
)
for row in resp.result.data_array or []:
print(row)
CLI surface varies by version. If a databricks subcommand (e.g. an experimental
tool, system-schemas, or external-lineage) is missing, fall back to the SQL warehouse
or the Python SDK shown above rather than guessing flags.
Best Practices
- Grant minimal access - Apply least privilege; grant at the narrowest securable that works
- Filter by date - System tables can be large; always use date filters
- Use appropriate retention - Check your workspace's retention settings
- Schedule reports - Create scheduled queries for regular monitoring
- Prefer SQL/SDK over experimental CLI - For anything beyond quick checks
- Connection passwords - after a rotation, run
ALTER CONNECTION with every option and the same secret(), or databricks connections update for a connection made without secret(). Azure Key Vault-backed scopes: the vault's access-policy model only, and a vault in another tenant needs the creating user to be able to create service principals there. Where the secret lives: references/8-connections-and-secrets.md
Related Skills
This skill owns Unity Catalog governance: access control, the privilege model,
external locations / storage credentials, securable DDL, fine-grained access, system
tables, and volumes. For adjacent concerns, use the sibling skill instead:
Roadmap (not yet covered — deferred to a later version)
These governance areas are intentionally out of scope for v0.4.0 and planned for later:
- Delta Sharing / Marketplace / Clean Rooms
- Lakehouse Federation (connection setup + foreign catalogs; connection passwords: Best Practice 6)
- ABAC / governed tags as policy
Resources
1---2name: databricks-unity-catalog3description: "Unity Catalog governance, access control, and observability. Use to grant or revoke access (GRANT/REVOKE), reason about the privilege model and ownership, set up row-level security and column masks, create external locations and storage credentials, define catalogs/schemas/tables/volumes, answer \"who can read this table\", query system tables (audit, lineage, billing), work with volume files in /Volumes/, and set up connection passwords (secret scopes, Azure Key Vault, AWS Secrets Manager)."4compatibility: Requires databricks CLI (>= v1.0.0)5metadata:6 source: https://skillmd.com/skills/stanfish06/databricks-unity-catalog7 version: "0.4.0"8parent: databricks-core9---1011# Unity Catalog1213Guidance for Unity Catalog **governance** — access control, the privilege model,14external locations, securable DDL, and fine-grained access — plus system tables and15volume file operations.1617> **Before running `databricks` CLI commands, confirm the CLI and the subcommand exist.**18> Run `databricks --version` — this skill assumes the unified CLI (**≥ v1.0.0**). Several19> subcommands shown here (`experimental aitools`, `system-schemas`, `external-lineage`,20> `grants`) vary by version or workspace availability; if one is missing or rejects a flag,21> fall back to the SQL form or the Python SDK rather than guessing. Each reference notes its22> own version floor where relevant.2324## When to Use This Skill2526Use this skill when:2728**Governance & access control (start here):**29- **Granting or revoking access** — `GRANT`/`REVOKE`, the UC privilege model, ownership (`ALTER … OWNER TO`), `SHOW GRANTS`, "who can read/write this table?"30- **Row- and column-level security** — row filters, column masks, dynamic views with `current_user()` / `is_account_group_member()`31- **External locations & storage credentials** — `CREATE STORAGE CREDENTIAL`, `CREATE EXTERNAL LOCATION`, backing external tables/volumes32- **Securable DDL & metadata** — creating/altering catalogs, schemas, managed vs external tables, views; comments, tags, table properties, ownership33- **Connection credentials & secrets**: a connection's password from a secret scope (`secret()`) and its rotation; where the secret lives3435**Observability & files:**36- Working with **volumes** (upload, download, list files in `/Volumes/`)37- Querying **lineage** (table dependencies, column-level lineage)38- Analyzing **audit logs** (who accessed what, permission changes)39- Monitoring **billing and usage** (DBU consumption, cost analysis)40- Tracking **compute resources** (cluster usage, warehouse metrics)41- Reviewing **job execution** (run history, success rates, failures)42- Analyzing **query performance** (slow queries, warehouse utilization)43- Profiling **data quality** (data profiling, drift detection, metric tables)4445## Reference Files4647| Topic | File | Description |48|-------|------|-------------|49| **Access Control** | [references/1-access-control.md](references/1-access-control.md) | Privilege model, securable hierarchy, GRANT/REVOKE, ownership, inheritance, `SHOW GRANTS` |50| **External Locations** | [references/2-external-locations.md](references/2-external-locations.md) | Storage credentials (AWS/Azure/GCP), external locations, validation |51| **Securables DDL** | [references/3-securables-ddl.md](references/3-securables-ddl.md) | CREATE/ALTER/DROP catalogs/schemas/tables/views, comments, tags, ownership |52| **Fine-Grained Access** | [references/4-fine-grained-access.md](references/4-fine-grained-access.md) | Row filters, column masks, dynamic views |53| System Tables | [references/5-system-tables.md](references/5-system-tables.md) | Lineage, audit, billing, compute, jobs, query history |54| Volumes | [references/6-volumes.md](references/6-volumes.md) | Volume file operations, permissions, best practices |55| Data Profiling | [references/7-data-profiling.md](references/7-data-profiling.md) | Data profiling, drift detection, profile metrics |56| Connections & Secrets | [references/8-connections-and-secrets.md](references/8-connections-and-secrets.md) | A connection's password and its rotation; where the secret lives |5758## Quick Start5960### Create Unity Catalog Objects (CLI)6162**Use `--json` for `create` commands.** Positional argument order differs per command and63has changed across CLI versions, so `--json` is the order-independent, version-stable form64shown throughout this skill.6566```bash67# Create a catalog68databricks catalogs create --json '{"name": "my_catalog"}'6970# Create a schema71databricks schemas create --json '{"name": "my_schema", "catalog_name": "my_catalog"}'7273# Create a managed volume74databricks volumes create --json '{75 "catalog_name": "my_catalog",76 "schema_name": "my_schema",77 "name": "my_volume",78 "volume_type": "MANAGED"79}'8081# List catalogs, schemas, volumes (read commands take simple positional args)82databricks catalogs list83databricks schemas list my_catalog84databricks volumes list my_catalog.my_schema85```8687Positional `create` args still work if you prefer them, but the order is **not** uniform88across commands — this is the per-command order (and the reason `--json` is recommended):8990| Command | Positional `create` order |91|---------|---------------------------|92| `databricks catalogs create` | `NAME` |93| `databricks schemas create` | `NAME CATALOG_NAME` |94| `databricks volumes create` | `CATALOG_NAME SCHEMA_NAME NAME VOLUME_TYPE` |9596> **CLI surface varies by version.** If a `databricks` subcommand or positional signature is97> missing in your install, prefer `--json`, the SQL form, or the Python SDK rather than98> guessing flags.99100### Volume File Operations (CLI)101102`databricks fs` requires the `dbfs:` scheme prefix even for UC Volume paths — without it the CLI treats the path as local filesystem and errors with `no such directory`.103104```bash105# List files in a volume106databricks fs ls dbfs:/Volumes/catalog/schema/volume/path/107108# Upload a directory's contents to a volume (-r copies contents, not the directory itself)109databricks fs cp -r --overwrite /tmp/data dbfs:/Volumes/catalog/schema/volume/dest110111# Download a file from a volume112databricks fs cp dbfs:/Volumes/catalog/schema/volume/file.csv /tmp/file.csv113114# Create a directory in a volume115databricks fs mkdirs dbfs:/Volumes/catalog/schema/volume/new_folder116```117118### Grant & Revoke Access119120`GRANT`/`REVOKE` is the core governance operation. See [references/1-access-control.md](references/1-access-control.md) for the full privilege model.121122```sql123-- Grant read access on a schema to a group124GRANT USE CATALOG ON CATALOG analytics TO `data_readers`;125GRANT USE SCHEMA ON SCHEMA analytics.gold TO `data_readers`;126GRANT SELECT ON SCHEMA analytics.gold TO `data_readers`;127128-- Who can access this table?129SHOW GRANTS ON TABLE analytics.gold.customers;130131-- Revoke132REVOKE SELECT ON SCHEMA analytics.gold FROM `data_readers`;133```134135### Enable System Tables Access136137```sql138-- Grant access to system tables139GRANT USE CATALOG ON CATALOG system TO `data_engineers`;140GRANT USE SCHEMA ON SCHEMA system.access TO `data_engineers`;141GRANT SELECT ON SCHEMA system.access TO `data_engineers`;142```143144### Common Queries145146```sql147-- Table lineage: What tables feed into this table?148SELECT source_table_full_name, source_column_name149FROM system.access.table_lineage150WHERE target_table_full_name = 'catalog.schema.table'151 AND event_date >= current_date() - 7;152153-- Audit: Recent permission changes154SELECT event_time, user_identity.email, action_name, request_params155FROM system.access.audit156WHERE action_name LIKE '%GRANT%' OR action_name LIKE '%REVOKE%'157ORDER BY event_time DESC158LIMIT 100;159160-- Billing: DBU usage by workspace161SELECT workspace_id, sku_name, SUM(usage_quantity) AS total_dbus162FROM system.billing.usage163WHERE usage_date >= current_date() - 30164GROUP BY workspace_id, sku_name;165```166167## Running SQL from the CLI168169> **`databricks experimental aitools tools query` is an experimental command.** The170> `experimental` namespace is not guaranteed to be stable across CLI versions and may be171> absent in your install. Prefer running system-table SQL from a **SQL warehouse** (SQL172> editor, scheduled query) or the **Python SDK** (`w.statement_execution.execute_statement`),173> or a **notebook**. Use the experimental CLI only for quick ad-hoc checks.174175> **Getting the IDs these examples use.** `WAREHOUSE_ID` — run `databricks warehouses list`176> (or copy it from a SQL warehouse's *Connection details* in the UI). `METASTORE_ID` (used in177> [references/5-system-tables.md](references/5-system-tables.md)) — `w.metastores.current().metastore_id`178> via the SDK, or the Catalog UI → metastore details.179180Experimental CLI form (convenience only):181182```bash183databricks experimental aitools tools query --warehouse WAREHOUSE_ID "184 SELECT source_table_full_name, target_table_full_name185 FROM system.access.table_lineage186 WHERE event_date >= current_date() - 7187"188```189190Stable SDK fallback (works on any CLI version):191192```python193from databricks.sdk import WorkspaceClient194195w = WorkspaceClient()196resp = w.statement_execution.execute_statement(197 warehouse_id="WAREHOUSE_ID",198 statement="""199 SELECT source_table_full_name, target_table_full_name200 FROM system.access.table_lineage201 WHERE event_date >= current_date() - 7202 LIMIT 100203 """,204)205for row in resp.result.data_array or []:206 print(row)207```208209> **CLI surface varies by version.** If a `databricks` subcommand (e.g. an `experimental`210> tool, `system-schemas`, or `external-lineage`) is missing, fall back to the SQL warehouse211> or the Python SDK shown above rather than guessing flags.212213## Best Practices2142151. **Grant minimal access** - Apply least privilege; grant at the narrowest securable that works2162. **Filter by date** - System tables can be large; always use date filters2173. **Use appropriate retention** - Check your workspace's retention settings2184. **Schedule reports** - Create scheduled queries for regular monitoring2195. **Prefer SQL/SDK over experimental CLI** - For anything beyond quick checks2206. **Connection passwords** - after a rotation, run `ALTER CONNECTION` with every option and the same `secret()`, or `databricks connections update` for a connection made without `secret()`. Azure Key Vault-backed scopes: the vault's access-policy model only, and a vault in another tenant needs the creating user to be able to create service principals there. Where the secret lives: [references/8-connections-and-secrets.md](references/8-connections-and-secrets.md)221222## Related Skills223224This skill owns Unity Catalog **governance**: access control, the privilege model,225external locations / storage credentials, securable DDL, fine-grained access, system226tables, and volumes. For adjacent concerns, use the sibling skill instead:227228- **databricks-core** (declared parent) — auth, profile selection, generic CLI, and catalog/table *exploration*229- **[databricks-metric-views](../databricks-metric-views/SKILL.md)** — metric view definitions / DDL (`WITH METRICS LANGUAGE YAML`)230- **[databricks-iceberg](../databricks-iceberg/SKILL.md)** — Managed Iceberg, External Iceberg Reads (fka Uniform), and **Iceberg REST Catalog (IRC) credential *vending*** for external engines — distinct from UC storage credentials (see [references/2-external-locations.md](references/2-external-locations.md))231- **[databricks-ml-training](../databricks-ml-training/SKILL.md)** — UC model registration and `@prod`/`@challenger` aliases232- **[databricks-vector-search](../databricks-vector-search/SKILL.md)** — Vector Search indexes233- **[databricks-pipelines](../databricks-pipelines/SKILL.md)**, **[databricks-jobs](../databricks-jobs/SKILL.md)**, **[databricks-lakeflow-connect](../databricks-lakeflow-connect/SKILL.md)** — *producing* tables via pipelines/jobs/managed ingestion234- **[databricks-lakebase](../databricks-lakebase/SKILL.md)** — Lakebase / synced tables (OLTP)235- **[databricks-ai-functions](../databricks-ai-functions/SKILL.md)** — AI functions such as `ai_mask` / `ai_classify` (AI *transforms*, **not** access control — see [references/4-fine-grained-access.md](references/4-fine-grained-access.md))236- **[databricks-aibi-dashboards](../databricks-aibi-dashboards/SKILL.md)** — AI/BI dashboards on UC data237- **[databricks-synthetic-data-gen](../databricks-synthetic-data-gen/SKILL.md)** — generating data stored in UC volumes238239### Roadmap (not yet covered — deferred to a later version)240241These governance areas are intentionally **out of scope for v0.4.0** and planned for later:242243- Delta Sharing / Marketplace / Clean Rooms244- Lakehouse Federation (connection setup + foreign catalogs; connection passwords: Best Practice 6)245- ABAC / governed tags as policy246247## Resources248249- [Unity Catalog Privileges & Securable Objects](https://docs.databricks.com/data-governance/unity-catalog/access-control/privileges-reference)250- [Unity Catalog System Tables](https://docs.databricks.com/admin/system-tables)251- [Audit Log Reference](https://docs.databricks.com/admin/account-settings/audit-logs)252- [Manage External Locations and Storage Credentials](https://docs.databricks.com/connect/unity-catalog/cloud-storage)