1---2name: databricks-iceberg3description: "Apache Iceberg tables on Databricks — Managed Iceberg tables, External Iceberg Reads (fka Uniform), Compatibility Mode, Iceberg REST Catalog (IRC), Iceberg v3, Snowflake interop, PyIceberg, OSS Spark, external engine access and credential vending. Use when creating Iceberg tables, enabling External Iceberg Reads (uniform) on Delta tables (including Streaming Tables and Materialized Views via compatibility mode), configuring external engines to read Databricks tables via Unity Catalog IRC, integrating with Snowflake catalog to read Foreign Iceberg tables"4compatibility: Requires databricks CLI (>= v1.0.0)5metadata:6 source: https://skillmd.com/skills/stanfish06/databricks-iceberg7 version: "0.1.0"8parent: databricks-core9---1011# Apache Iceberg on Databricks1213Databricks provides multiple ways to work with Apache Iceberg: native managed Iceberg tables, UniForm for Delta-to-Iceberg interoperability, and the Iceberg REST Catalog (IRC) for external engine access.1415---1617## Critical Rules (always follow)1819- **MUST** use Unity Catalog — all Iceberg features require UC-enabled workspaces20- **MUST NOT** install an Iceberg library into Databricks Runtime (DBR includes built-in Iceberg support; adding a library causes version conflicts)21- **MUST NOT** set `write.metadata.path` or `write.metadata.previous-versions-max` — Databricks manages metadata locations automatically; overriding causes corruption22- **MUST** determine which Iceberg pattern fits the use case before writing code — see the [When to Use](#when-to-use) section below23- **MUST** know that both `PARTITIONED BY` and `CLUSTER BY` produce the same Iceberg metadata for external engines — UC maintains an Iceberg partition spec with partition fields corresponding to the clustering keys, so external engines reading via IRC see a partitioned Iceberg table (not Hive-style, but proper Iceberg partition fields) and can prune on those fields; internally UC uses those fields as liquid clustering keys; the only differences between the two syntaxes are: (1) `PARTITIONED BY` is standard Iceberg DDL (any engine can create the table), while `CLUSTER BY` is DBR-only DDL; (2) `PARTITIONED BY` **auto-handles** DV/row-tracking properties, while `CLUSTER BY` requires manual TBLPROPERTIES on v224- **MUST NOT** use expression-based partition transforms (`bucket()`, `years()`, `months()`, `days()`, `hours()`) with `PARTITIONED BY` on managed Iceberg tables — only plain column references are supported; expression transforms cause errors25- **MUST** disable deletion vectors and row tracking when using `CLUSTER BY` on Iceberg v2 tables — set `'delta.enableDeletionVectors' = false` and `'delta.enableRowTracking' = false` in TBLPROPERTIES (Iceberg v3 handles this automatically; `PARTITIONED BY` handles this automatically on both v2 and v3)2627---2829## Key Concepts3031| Concept | Summary |32|---------|---------|33| **Managed Iceberg Table** | Native Iceberg table created with `USING ICEBERG` — full read/write in Databricks and via external Iceberg engines |34| **External Iceberg Reads (Uniform)** | Delta table that auto-generates Iceberg metadata — read as Iceberg externally, write as Delta internally |35| **Compatibility Mode** | UniForm variant for streaming tables and materialized views in SDP pipelines |36| **Iceberg REST Catalog (IRC)** | Unity Catalog's built-in REST endpoint implementing the Iceberg REST Catalog spec — lets external engines (Spark, PyIceberg, Snowflake) access UC-managed Iceberg data |37| **Iceberg v3** | Next-gen format (DBR 18 LTS+) — deletion vectors, VARIANT type, row lineage |3839---4041## Quick Start4243### Create a Managed Iceberg Table4445```sql46-- No clustering47CREATE TABLE my_catalog.my_schema.events48USING ICEBERG49AS SELECT * FROM raw_events;5051-- PARTITIONED BY (recommended for cross-platform): standard Iceberg syntax, works on EMR/OSS Spark/Trino/Flink52-- auto-disables DVs and row tracking — no TBLPROPERTIES needed on v2 or v353CREATE TABLE my_catalog.my_schema.events54USING ICEBERG55PARTITIONED BY (event_date)56AS SELECT * FROM raw_events;5758-- CLUSTER BY on Iceberg v2 (DBR-only syntax): must manually disable DVs and row tracking59CREATE TABLE my_catalog.my_schema.events60USING ICEBERG61TBLPROPERTIES (62 'delta.enableDeletionVectors' = false,63 'delta.enableRowTracking' = false64)65CLUSTER BY (event_date)66AS SELECT * FROM raw_events;6768-- CLUSTER BY on Iceberg v3 (DBR-only syntax): no TBLPROPERTIES needed69CREATE TABLE my_catalog.my_schema.events70USING ICEBERG71TBLPROPERTIES ('format-version' = '3')72CLUSTER BY (event_date)73AS SELECT * FROM raw_events;74```7576### Enable UniForm on an Existing Delta Table7778```sql79ALTER TABLE my_catalog.my_schema.customers80SET TBLPROPERTIES (81 'delta.columnMapping.mode' = 'name',82 'delta.enableIcebergCompatV2' = 'true',83 'delta.universalFormat.enabledFormats' = 'iceberg'84);85```8687---8889## Read/Write Capability Matrix9091| Table Type | Databricks Read | Databricks Write | External IRC Read | External IRC Write |92|------------|:-:|:-:|:-:|:-:|93| Managed Iceberg (`USING ICEBERG`) | Yes | Yes | Yes | Yes |94| Delta + UniForm | Yes (as Delta) | Yes (as Delta) | Yes (as Iceberg) | No |95| Delta + Compatibility Mode | Yes (as Delta) | Yes | Yes (as Iceberg) | No |9697---9899## Reference Files100101| File | Summary | Keywords |102|------|---------|----------|103| [references/1-managed-iceberg-tables.md](references/1-managed-iceberg-tables.md) | Creating and managing native Iceberg tables — DDL, DML, Liquid Clustering, Predictive Optimization, Iceberg v3, limitations | CREATE TABLE USING ICEBERG, CTAS, MERGE, time travel, deletion vectors, VARIANT |104| [references/2-uniform-and-compatibility.md](references/2-uniform-and-compatibility.md) | Making Delta tables readable as Iceberg — UniForm for regular tables, Compatibility Mode for streaming tables and MVs | UniForm, universalFormat, Compatibility Mode, streaming tables, materialized views, SDP |105| [references/3-iceberg-rest-catalog.md](references/3-iceberg-rest-catalog.md) | Exposing Databricks tables to external engines via the IRC endpoint — auth, credential vending, IP access lists | IRC, REST Catalog, credential vending, EXTERNAL USE SCHEMA, PAT, OAuth |106| [references/4-snowflake-interop.md](references/4-snowflake-interop.md) | Bidirectional Snowflake-Databricks integration — catalog integration, foreign catalogs, vended credentials | Snowflake, catalog integration, external volume, vended credentials, REFRESH_INTERVAL_SECONDS |107| [references/5-external-engine-interop.md](references/5-external-engine-interop.md) | Connecting PyIceberg, OSS Spark, AWS EMR, Apache Flink, and Kafka Connect via IRC | PyIceberg, OSS Spark, EMR, Flink, Kafka Connect, pyiceberg.yaml |108109---110111## When to Use112113- **Creating a new Iceberg table** → [references/1-managed-iceberg-tables.md](references/1-managed-iceberg-tables.md)114- **Making an existing Delta table readable as Iceberg** → [references/2-uniform-and-compatibility.md](references/2-uniform-and-compatibility.md)115- **Making a streaming table or MV readable as Iceberg** → [references/2-uniform-and-compatibility.md](references/2-uniform-and-compatibility.md) (Compatibility Mode section)116- **Choosing between Managed Iceberg vs UniForm vs Compatibility Mode** → decision table in [references/2-uniform-and-compatibility.md](references/2-uniform-and-compatibility.md)117- **Exposing Databricks tables to external engines via REST API** → [references/3-iceberg-rest-catalog.md](references/3-iceberg-rest-catalog.md)118- **Integrating Databricks with Snowflake (either direction)** → [references/4-snowflake-interop.md](references/4-snowflake-interop.md)119- **Connecting PyIceberg, OSS Spark, Flink, EMR, or Kafka** → [references/5-external-engine-interop.md](references/5-external-engine-interop.md)120121---122123## Common Issues124125| Issue | Solution |126|-------|----------|127| **No Change Data Feed (CDF)** | CDF is not supported on managed Iceberg tables. Use Delta + UniForm if you need CDF. |128| **UniForm async delay** | Iceberg metadata generation is asynchronous. After a write, there may be a brief delay before external engines see the latest data. Check status with `DESCRIBE EXTENDED table_name`. |129| **Compression codec change** | Managed Iceberg tables use `zstd` compression by default (not `snappy`). Older Iceberg readers that don't support zstd will fail. Verify reader compatibility or set `write.parquet.compression-codec` to `snappy`. |130| **Snowflake 1000-commit limit** | Snowflake's Iceberg catalog integration can only see the last 1000 Iceberg commits. High-frequency writers must compact metadata or Snowflake will lose visibility of older data. |131| **Deletion vectors with UniForm** | UniForm requires deletion vectors to be disabled (`delta.enableDeletionVectors = false`). If your table has deletion vectors enabled, disable them before enabling UniForm. |132| **No shallow clone for Iceberg** | `SHALLOW CLONE` is not supported for Iceberg tables. Use `DEEP CLONE` or `CREATE TABLE ... AS SELECT` instead. |133| **Version mismatch with external engines** | Ensure external engines use an Iceberg library version compatible with the format version of your tables. Iceberg v3 tables require Iceberg library 1.9.0+. |134135---136137## Related Skills138139- **[databricks-unity-catalog](../databricks-unity-catalog/SKILL.md)** — catalog/schema management, governance, system tables140- **databricks-pipelines** — SDP pipelines (streaming tables, materialized views with Compatibility Mode)141- **[databricks-python-sdk](../databricks-python-sdk/SKILL.md)** — Python SDK and REST API for Databricks operations142- **[databricks-dbsql](../databricks-dbsql/SKILL.md)** — SQL warehouse features, query patterns143144---145146## Resources147148- **[Iceberg Overview](https://docs.databricks.com/iceberg/)** — main hub for Iceberg on Databricks149- **[UniForm](https://docs.databricks.com/delta/iceberg-reads)** — Delta Universal Format150- **[Iceberg REST Catalog](https://docs.databricks.com/external-access/iceberg)** — IRC endpoint and external engine access151- **[Compatibility Mode](https://docs.databricks.com/external-access/compatibility-mode)** — UniForm for streaming tables and MVs152- **[Iceberg v3](https://docs.databricks.com/iceberg/iceberg-v3)** — next-gen format features153- **[Foreign Tables](https://docs.databricks.com/tables/foreign)** — reading external catalog data