Iceberg Best Practices

Apache Iceberg Best Practices

Core Principles

• Use hidden partitioning with transforms — never create explicit partition columns
• Run table maintenance regularly — expire snapshots, compact files, rewrite manifests
• Use Parquet with ZSTD compression for all data files
• Configure sort order on frequently filtered columns
• Use a production catalog (REST, Glue, or Hive Metastore) — never Hadoop catalog
• Set state TTL / snapshot retention to prevent unbounded metadata growth
• Design for schema evolution from day one — Iceberg tracks columns by ID, not name

Table Design and Schema Evolution (CRITICAL)

schema-evolution-safety

Iceberg schema evolution is safe — use it instead of recreating tables.

Every column gets a unique ID. Renaming, reordering, adding, or dropping columns are metadata-only operations that never rewrite data files.

Safe operations (metadata-only):

• Add optional column
• Drop column (data files untouched; column excluded from reads)
• Rename column (tracked by ID — historical data readable under new name)
• Reorder columns
• Widen type (int -> long, float -> double, decimal(P,S) -> decimal(P',S) where P' > P)
• Make required field optional

Not supported:

• Narrowing types (long -> int)
• Incompatible type changes (string -> int)
• Making optional field required

schema-type-choices

Choose types carefully — they are hard to change later.

schema-best-practices

Design schemas for evolution from day one.

• Add new columns as optional with documented defaults
• Use COMMENT ON COLUMN to track field semantics
• Coordinate with downstream consumers before dropping columns
• Test schema changes in a branch or test environment before production

Partitioning (CRITICAL)

partition-hidden

Use hidden partitioning with transforms — never create explicit partition columns.

Iceberg applies partition transforms automatically. Users query raw columns and the engine applies partition pruning transparently. No synthetic partition columns needed.

CREATE TABLE events (
    event_id STRING,
    event_time TIMESTAMP,
    user_id BIGINT,
    event_type STRING
) USING iceberg
PARTITIONED BY (day(event_time), bucket(16, user_id));

Available transforms: year(ts), month(ts), day(ts), hour(ts), bucket(N, col), truncate(L, col), identity(col).

partition-transform-selection

Choose partition transforms based on data volume.

For non-time columns, use bucket(N, col) where N produces partitions of 100 MB–1 GB each.

partition-sizing

Target partition sizes of 100 MB–1 GB per partition.

• Avoid more than ~10,000 total partitions (metadata overhead)
• Avoid fewer than ~10 partitions (limits parallelism)
• If partitions produce files < 25 MB, use coarser partitioning or compaction
• If partitions produce files > 2 GB, use finer partitioning

partition-evolution

Iceberg supports changing partition schemes without rewriting data.

Old data retains the old layout; new data uses the new layout. The query planner handles both transparently.

-- Evolve from monthly to daily as data volume grows
ALTER TABLE db.events SET PARTITION SPEC (day(event_time), bucket(16, user_id));

partition-anti-patterns

Avoid these partitioning mistakes.

• identity(high_cardinality_column) — creates one partition per unique value
• hour(ts) on low-volume tables — thousands of tiny files
• Three-level partitioning — almost never needed; keep fanout manageable
• Explicit partition columns (Hive-style) — defeats Iceberg's hidden partitioning

Sort Order (HIGH)

sort-order-configuration

Set sort order on frequently filtered columns to enable min/max metadata filtering.

ALTER TABLE db.events WRITE ORDERED BY (tenant_id ASC NULLS LAST, event_time ASC);

• Sort by columns most commonly used in WHERE clauses (after partition columns)
• Put low-cardinality filter columns first
• Sort order applies only to newly written files; run rewrite_data_files with sort strategy to apply to existing data

sort-order-zorder

Use Z-order when queries filter on multiple columns with roughly equal selectivity.

Z-order interleaves bits of multiple columns for balanced multi-dimensional clustering. Linear sort is better when one column dominates filter patterns.

CALL catalog.system.rewrite_data_files(
  table => 'db.events',
  strategy => 'zorder',
  sort_order => 'tenant_id,event_type'
)

File Format (HIGH)

format-parquet-zstd

Use Parquet with ZSTD compression. There is rarely a reason to deviate.

ALTER TABLE db.events SET TBLPROPERTIES (
  'write.format.default' = 'parquet',
  'write.parquet.compression-codec' = 'zstd'
);

Key tuning properties:

• write.parquet.compression-codec: zstd (recommended), snappy, gzip, lz4
• write.parquet.compression-level: 1–3 for ZSTD (good balance)
• write.parquet.row-group-size-bytes: 134217728 (128 MB default)
• write.parquet.dict-size-bytes: 2097152 (2 MB; increase for high-cardinality dictionary columns)

Catalog Configuration (CRITICAL)

catalog-selection

Choose a production catalog. Never use Hadoop catalog in production.

catalog-rest

Use REST catalog for multi-engine deployments.

spark.sql.catalog.my_catalog=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.my_catalog.type=rest
spark.sql.catalog.my_catalog.uri=http://catalog-host:port/api/v1
spark.sql.catalog.my_catalog.warehouse=s3://bucket/warehouse

REST catalog supports credential vending — engines get short-lived credentials for storage access.

catalog-caching

Enable catalog caching to reduce metadata lookups.

spark.sql.catalog.my_catalog.cache-enabled=true
spark.sql.catalog.my_catalog.cache.expiration-interval-ms=300000

Write Optimization (HIGH)

write-target-file-size

Target file sizes of 256 MB–1 GB.

ALTER TABLE db.events SET TBLPROPERTIES (
  'write.target-file-size-bytes' = '536870912'  -- 512 MB
);

• High-query-frequency tables: 256–512 MB (more parallelism)
• Archival/cold tables: 512 MB–1 GB (fewer files, less metadata)
• Never below 32 MB (too many small files) or above 2 GB (too little parallelism)

write-distribution-mode

Choose write distribution mode based on table structure.

ALTER TABLE db.events SET TBLPROPERTIES (
  'write.distribution-mode' = 'hash'
);

write-streaming-flink

For Flink streaming, set checkpoint interval to 1–5 minutes and target 128 MB files.

ALTER TABLE db.events SET TBLPROPERTIES (
  'write.target-file-size-bytes' = '134217728'  -- 128 MB for streaming
);

10-second checkpoint intervals create many tiny files. Use longer intervals and rely on compaction.

write-commit-retry

Increase commit retries for high-contention environments.

commit.retry.num-retries=4
commit.retry.min-wait-ms=100
commit.manifest-merge.enabled=true
commit.manifest.target-size-bytes=8388608

Read Optimization (HIGH)

read-predicate-pushdown

Always filter on partition columns first, then sorted columns.

Iceberg pushes predicates down to three levels:

1. Partition pruning — skips entire partitions
2. Manifest filtering — min/max per file from manifest metadata
3. Row-group filtering — min/max within Parquet row groups

Use explicit equality and range predicates. Avoid UDFs or complex expressions that prevent pushdown.

read-column-pruning

Select only needed columns — never SELECT * on wide tables.

Iceberg + Parquet skips entire column chunks for unselected columns. Wide tables (100+ columns) benefit enormously.

read-metadata-metrics

Configure column-level metrics for optimal metadata filtering.

ALTER TABLE db.events SET TBLPROPERTIES (
  'write.metadata.metrics.default' = 'truncate(16)',
  'write.metadata.metrics.column.tenant_id' = 'full',
  'write.metadata.metrics.column.raw_payload' = 'none'
);

• full: complete min/max stats — for columns always filtered on
• truncate(N): first N bytes of min/max — default, good for most strings
• none: no stats — for columns never filtered on (reduces metadata size)

read-split-size

Tune split size for read parallelism.

ALTER TABLE db.events SET TBLPROPERTIES (
  'read.split.target-size' = '134217728'  -- 128 MB default
);

Decrease for more parallelism on large clusters; increase for small clusters.

read-vectorized

Enable vectorized reading in Spark.

spark.sql.iceberg.vectorization.enabled=true

Processes data in columnar batches rather than row-by-row — significant speedup for scan-heavy queries.

Table Maintenance (CRITICAL)

maintenance-expire-snapshots

Expire snapshots daily. This is the most important maintenance task.

CALL catalog.system.expire_snapshots(
  table => 'db.events',
  older_than => TIMESTAMP '2024-01-01 00:00:00',
  retain_last => 10
);

Without expiration, metadata files grow unboundedly, slowing down all query planning. Keep enough snapshots for your time travel needs (default retention: 5 days).

maintenance-compaction

Run binpack compaction daily (hourly for streaming tables).

CALL catalog.system.rewrite_data_files(
  table => 'db.events',
  strategy => 'binpack',
  options => map(
    'target-file-size-bytes', '536870912',
    'min-file-size-bytes', '67108864',
    'max-file-size-bytes', '1073741824',
    'min-input-files', '5',
    'partial-progress.enabled', 'true',
    'partial-progress.max-commits', '10'
  )
);

Run binpack daily/hourly. Run sort weekly during off-peak hours.

maintenance-rewrite-manifests

Rewrite manifests weekly to improve query planning performance.

CALL catalog.system.rewrite_manifests(table => 'db.events');

Consolidates many small manifest files into fewer, larger ones. Run after heavy write operations.

maintenance-orphan-files

Remove orphan files weekly or monthly.

CALL catalog.system.remove_orphan_files(
  table => 'db.events',
  older_than => TIMESTAMP '2024-06-01 00:00:00',
  dry_run => true  -- preview first
);

Use older_than > max expected write duration to avoid deleting in-progress writes (default: 3 days).

maintenance-schedule

Recommended maintenance schedule.

Time Travel and Snapshots (MEDIUM)

timetravel-queries

Query historical data by snapshot ID or timestamp.

SELECT * FROM db.events VERSION AS OF 1234567890;
SELECT * FROM db.events TIMESTAMP AS OF '2024-06-15 10:00:00';

Time travel only works for snapshots that haven't been expired. Retain enough snapshots for your incident response SLA.

timetravel-rollback

Rollback is a metadata-only operation (instant).

CALL catalog.system.rollback_to_snapshot('db.events', 1234567890);
CALL catalog.system.rollback_to_timestamp('db.events', TIMESTAMP '2024-06-15 10:00:00');

Does not delete data files — the rolled-back-to snapshot's files are still present.

timetravel-tags

Use tags to protect important snapshots from expiration.

ALTER TABLE db.events CREATE TAG release_2024_q2 AS OF VERSION 1234567890 RETAIN 365 DAYS;

Tagged snapshots are immune from expire_snapshots for the retention period.

Merge-on-Read vs Copy-on-Write (HIGH)

mor-vs-cow

Choose based on read/write ratio.

-- Copy-on-Write (default, recommended for most tables)
ALTER TABLE db.events SET TBLPROPERTIES (
  'write.delete.mode' = 'copy-on-write',
  'write.update.mode' = 'copy-on-write',
  'write.merge.mode' = 'copy-on-write'
);

-- Merge-on-Read (for streaming/CDC tables)
ALTER TABLE db.events SET TBLPROPERTIES (
  'write.delete.mode' = 'merge-on-read',
  'write.update.mode' = 'merge-on-read',
  'write.merge.mode' = 'merge-on-read'
);

Positional vs equality deletes:

• Positional deletes reference file path + row position — efficient for engines to apply, used by Spark
• Equality deletes reference column values to match — more flexible but more expensive at read time
• Positional deletes are preferred; Iceberg v2 supports both

Critical for MoR: Run compaction regularly to merge delete files into base data files. If a data file has >10 associated delete files, compaction is overdue.

Multi-Engine Integration (HIGH)

integration-spark

Spark has the most mature Iceberg integration.

spark.sql.catalog.my_catalog=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.my_catalog.type=rest
spark.sql.catalog.my_catalog.uri=http://catalog-host:port/api/v1

• Use Spark for all maintenance procedures (rewrite_data_files, expire_snapshots, etc.)
• Enable vectorized reads: spark.sql.iceberg.vectorization.enabled=true
• Use DataFrame.writeTo("catalog.db.table").append() or .overwritePartitions() for writes
• spark.sql.shuffle.partitions affects output file count

integration-flink

Flink is best for streaming ingestion into Iceberg.

• Set checkpoint interval to control commit frequency (1–5 minutes)
• write.upsert.enabled=true for CDC upsert semantics
• Flink writes smaller files by nature (streaming) — rely on compaction to consolidate
• Target 128 MB file sizes for streaming

integration-trino

Trino is excellent for interactive analytics and ad-hoc queries.

• Supports reads, writes, schema evolution, time travel, hidden partitioning
• Does not support all maintenance procedures — use Spark for full maintenance
• Set iceberg.file-format=PARQUET and iceberg.compression-codec=ZSTD

integration-cross-engine

Use a shared catalog for multi-engine access.

• All engines read/write the same Iceberg format — true interoperability
• Ensure all engines use compatible Iceberg library versions
• Avoid engine-specific table properties when multi-engine access is needed

Branching and Tagging (MEDIUM)

branching-wap

Use branches for Write-Audit-Publish (WAP) workflows.

-- Create a branch
ALTER TABLE db.events CREATE BRANCH audit_branch;

-- Write to branch (Spark)
SET spark.wap.branch = audit_branch;
INSERT INTO db.events VALUES (...);

-- Read from branch
SELECT * FROM db.events VERSION AS OF 'audit_branch';

-- Fast-forward main after validation
CALL catalog.system.fast_forward('db.events', 'main', 'audit_branch');

branching-nessie

Use Nessie for catalog-level branching across multiple tables.

Nessie provides Git-like branching across the entire catalog. Create a branch, modify multiple tables, merge atomically. Enables data CI/CD with merge conflict detection.

branching-cleanup

Set retention on branches and tags. Clean up stale branches.

ALTER TABLE db.events DROP BRANCH old_branch;

Unmanaged branches and tags cause unbounded metadata growth.

Common Anti-Patterns