Lakehouse Table Format Engineering
Overview
Use this skill when the storage layer itself is an architecture decision. It helps agents reason about Iceberg, Delta, Hudi, table evolution, partitioning, compaction, snapshot behavior, and how batch and streaming jobs interact with shared lakehouse tables.
When to Use
- choosing between
Iceberg, Delta, and Hudi
- designing lakehouse datasets with update or merge behavior
- changing partition strategy, snapshot retention, or compaction behavior
- integrating
Spark, Flink, or warehouse readers with shared table formats
- implementing CDC-style or incremental lakehouse pipelines
Do not use this for unmanaged flat-file datasets that are not intended to behave like governed tables.
Workflow
Define the table contract.
Include:
- grain and keys
- mutation model
- read patterns
- retention and snapshot policy
- compatibility requirements across engines
Pick the table format intentionally.
Iceberg: strong open-table interoperability and metadata-driven planningDelta: strong lakehouse ergonomics and transactional patterns in Databricks-centered stacksHudi: strong incremental and record-level update use cases
Design physical layout for lifecycle, not only day-one queries.
Consider:
- partition evolution
- small-file control
- compaction
- metadata growth
- merge and delete behavior
Align compute engines with the table behavior.
Batch and streaming writers must not conflict silently on checkpoints, commits, or schema changes.
Define maintenance and recovery operations.
Table formats require operational housekeeping, not just one-time creation.
Common Rationalizations
| Rationalization |
Reality |
| "A table format is just a storage detail." |
Format choice affects interoperability, maintenance, mutation semantics, and long-term cost. |
| "We can decide compaction later." |
Small files and metadata bloat become operational pain quickly. |
| "Any engine can read and write the table the same way." |
Cross-engine support varies, especially for advanced write and mutation behavior. |
Red Flags
- format selection is based only on familiarity
- mutation and delete semantics are undefined
- snapshot retention and cleanup are ignored
- batch and stream writers target the same table without coordination
- interoperability requirements are discovered after adoption
Verification
1---2name: lakehouse-table-format-engineering3description: Guides agents through lakehouse table design and open table format decisions. Use when designing or changing Iceberg, Delta, Hudi, partitioning, schema evolution, compaction, or batch and streaming interoperability.4---56# Lakehouse Table Format Engineering78## Overview910Use this skill when the storage layer itself is an architecture decision. It helps agents reason about `Iceberg`, `Delta`, `Hudi`, table evolution, partitioning, compaction, snapshot behavior, and how batch and streaming jobs interact with shared lakehouse tables.1112## When to Use1314- choosing between `Iceberg`, `Delta`, and `Hudi`15- designing lakehouse datasets with update or merge behavior16- changing partition strategy, snapshot retention, or compaction behavior17- integrating `Spark`, `Flink`, or warehouse readers with shared table formats18- implementing CDC-style or incremental lakehouse pipelines1920Do not use this for unmanaged flat-file datasets that are not intended to behave like governed tables.2122## Workflow23241. Define the table contract.25 Include:26 - grain and keys27 - mutation model28 - read patterns29 - retention and snapshot policy30 - compatibility requirements across engines31322. Pick the table format intentionally.33 - `Iceberg`: strong open-table interoperability and metadata-driven planning34 - `Delta`: strong lakehouse ergonomics and transactional patterns in Databricks-centered stacks35 - `Hudi`: strong incremental and record-level update use cases36373. Design physical layout for lifecycle, not only day-one queries.38 Consider:39 - partition evolution40 - small-file control41 - compaction42 - metadata growth43 - merge and delete behavior44454. Align compute engines with the table behavior.46 Batch and streaming writers must not conflict silently on checkpoints, commits, or schema changes.47485. Define maintenance and recovery operations.49 Table formats require operational housekeeping, not just one-time creation.5051## Common Rationalizations5253| Rationalization | Reality |54| --- | --- |55| "A table format is just a storage detail." | Format choice affects interoperability, maintenance, mutation semantics, and long-term cost. |56| "We can decide compaction later." | Small files and metadata bloat become operational pain quickly. |57| "Any engine can read and write the table the same way." | Cross-engine support varies, especially for advanced write and mutation behavior. |5859## Red Flags6061- format selection is based only on familiarity62- mutation and delete semantics are undefined63- snapshot retention and cleanup are ignored64- batch and stream writers target the same table without coordination65- interoperability requirements are discovered after adoption6667## Verification6869- [ ] Table format choice is tied to interoperability and workload needs70- [ ] Mutation, partitioning, retention, and compaction rules are explicit71- [ ] Engine compatibility assumptions are documented72- [ ] Maintenance and recovery procedures exist for the table lifecycle
vaquarkhan