Lakehouse Standards

Lakehouse standards (open table format)

Criteria verified as of August 2026. Re-verify on the web before committing to anything (§8).

Hard premise: a lakehouse is not a Parquet directory with a marketing name. It is a table format —a versioned metadata tree on top of a bucket— that buys you ACID, schema evolution, time travel and concurrent read/write, and is paid for in catalog, compaction and permanent maintenance. If you are not going to run that maintenance, you do not have a lakehouse: you have a swamp with snapshots.

1. Scope and triggers

Applies to the table format and to everything decided from it: format choice, catalog, metadata anatomy, partitioning and its evolution, table maintenance, write concurrency, query engines over the same data, update and delete strategy, and table-level governance/security.

Triggers: metadata.json, snap-*.avro, manifest-list, .metadata/, _delta_log/, .hoodie/, format-version, iceberg.catalog.type, catalog-impl, warehouse=, spark.sql.catalog.*, USING iceberg / USING delta, CALL ... system.rewrite_data_files, expire_snapshots, remove_orphan_files, rewrite_manifests, VACUUM, OPTIMIZE, ZORDER, MERGE INTO, FOR SYSTEM_TIME AS OF / VERSION AS OF, pyiceberg, delta-rs/deltalake, iceberg-rest, polaris, nessie, gravitino, lakekeeper, unitycatalog, glue_catalog, hive metastore, thrift://, s3tables, and the phrases that give the domain away: "the query reads 50,000 files", "the bucket keeps growing", "two jobs collide when writing the table", "I need to see the table as it was on Tuesday", "we changed the partitioning and now everything has to be rewritten", "deleting a user does not apply to the history".

Not applicable: see

• data-engineering-standards (sister; cut rule already written there and mirrored here without changes): the file format —Parquet, file size, compression, file typing— is theirs; the table format —Iceberg/Delta/Hudi, catalog, snapshots, compaction, hidden partitioning— belongs here. If the decision is made by the table format, it belongs here; if it is made by the process that writes, it is theirs. Corollary: ingestion, orchestration, job idempotency, the backfill and the small-files problem at source are theirs; compaction as a table operation belongs here.
• data-warehouse-modeling-standards (thin boundary, declared on both sides): the time travel of a table format gives you back the table as it was; an SCD type 2 tells you how the business entity was. They are different things: the former is an infrastructure capability for recovery and technical audit, with retention of days or weeks; the latter is a modelling decision the business queries indefinitely. Replacing an SCD2 with time travel is forbidden there and it is forbidden here (§7). Grain, facts, dimensions and metrics are theirs; this skill models nothing.
• data-platform-standards (mother): PostgreSQL, Redis/Valkey, Kafka as an engine (topic partitions, retention, schema registry), engine backups and PITR, encryption at rest. Its guiding principle —one store per need, not per fashion— is inherited here without exception and is precisely the filter of §2.1: this skill does not authorise a lakehouse, it decides how to do a already-justified one well.
• object-storage-standards: the substrate is theirs — buckets, bucket policy, storage classes and their retrieval latency, Object Lock and legal hold, object versioning, lifecycle, multipart, AbortIncompleteMultipartUpload, checksums, cost per request and per egress, and the antipattern of mounting S3 as a disk. Here, what key structure and what request pattern a table format produces on top, and what it demands from the bucket. Arbitration rule: if the question is about the bucket, it is theirs; if it is about the table, it belongs here. Cross warning: Object Lock in compliance mode and a table with expire_snapshots are incompatible in practice — maintenance will not be able to delete anything (§5).
• streaming-cdc-standards (sister; boundary declared on both sides): how changes are captured and how they are processed in motion is theirs —transaction log, Debezium, initial snapshot, replication slots, delivery semantics, per-partition ordering, windows and watermarks—. Here, what happens to the table when those changes land: copy-on-write versus merge-on-read, cost of MERGE/DELETE, delete files and deletion vectors, flow-induced compaction. CDC is a typical lakehouse source, not the only one: the lakehouse is fed just as well in batch; and CDC feeds other destinations just as well (a replica, a cache, a search index). Neither skill presupposes the other.
• microservices-architecture-standards: outbox, sagas, domain events and per-service data ownership are theirs. A domain event is not a row of an analytical table.
• privacy-engineering-standards: retention, minimisation, pseudonymisation, crypto-shredding and data subject rights are theirs. Here only the technical constraint imposed by an immutable format with history and how deletion is executed inside it (§5.3).
• data-governance-quality-standards (already on disk): business and metadata catalog (DataHub, OpenMetadata, Collibra, Atlan…), ownership and stewardship, glossary, data contracts, lineage as a programme and quality as a discipline. Mandatory boundary because of the "catalog" homonym: the catalog in this document (Polaris, Glue, Unity, Nessie, Gravitino) is a runtime component that resolves the table pointer, sequences the commits and issues credentials; theirs is a documentary and governance inventory. Cut rule: if the component is on the path of a query, it belongs here; if it is on the path of a person who searches for or governs a dataset, it is theirs. Unity Catalog appears in both lists precisely because it tries to be both things — its Iceberg catalog facet belongs here.
• analytics-bi-standards: the BI tool and consumption.
• message-brokers-standards: the broker as a component.
• nosql-standards, graph-db-standards, vector-db-standards, search-engines-standards, timeseries-db-standards, oracle-dba-standards, sqlserver-dba-standards, mysql-mariadb-dba-standards: other stores, with their own criteria.
• aws-standards/azure-standards/gcp-standards: Glue, S3 Tables, Databricks, BigQuery, Fabric/OneLake as managed services —provisioning, IAM, network, billing—; here the format and catalog criteria that apply equally in all three.
• backup-recovery-standards and bcdr-standards: time travel is not a backup (§7). What is copied, with what chain and how the restore is proven is theirs.
• identity-access-management-standards (the identity the catalog authenticates), secrets-management-standards (the credentials the catalog vends), observability-standards, sre-practice-standards, iac-standards, cicd-standards, kubernetes-standards, grc-compliance-standards, mlops-standards, rag-standards, python-standards (pyiceberg, delta-rs), jvm-spring-standards (the JVM and its tuning under the engines), scala-standards (the Scala written inside a Spark or Flink job: here the table format, the catalog and maintenance; there the code).
• sql-standards (the SQL language). Boundary named because MERGE appears in both: here it is an operation on the table —copy-on-write versus merge-on-read cost, delete files, compaction, snapshots it leaves behind—; there it is a clause that is written —match conditions, non-determinism when the source duplicates rows, per-dialect alternatives when the engine does not have it—. Neither duplicates the other.

2. Default decisions

Verify version, licence, governance and real adoption status on the web before committing to anything in a real project (§8). This subsector moved a lot in 2025-2026 and a stale snapshot here costs a migration.

2.1 The starting decision: do you really need a lakehouse?

"Lakehouse" is one of the two most over-sold labels in the sector (the other is handled by streaming-cdc-standards). Before adopting it, exhaust this ladder. Every rung avoided is infrastructure you do not operate:

What a Parquet directory really does not give you —and it is the only list that justifies the jump—:

1. Atomicity and isolation: a reader never sees a half-finished write; a failed INSERT OVERWRITE does not leave the table in an intermediate state.
2. Row-level deletes and updates without rewriting the whole partition by hand.
3. Safe schema evolution (add, rename, reorder, change type within rules) by column ID, not by position nor by file name.
4. Time travel and rollback to an earlier snapshot after a bad write.
5. Concurrent writing with optimistic control, instead of "last writer wins".
6. Planning without listing the bucket: the metadata knows which files exist and what ranges they contain; listing S3 stops being the critical operation.
7. Partitioning decoupled from the path: it can be changed without rewriting the history.

What it costs, and this has to be signed off up front: a catalog that has to be deployed, authenticated, backed up and made HA; a periodic maintenance job (compaction, expiration, orphans) with its compute and its bill; a new failure mode (the table corrupts if you delete files underneath the catalog); and one more piece to update and explain to whoever replaces you.

Rule: if nobody on the team can name who runs compaction and at what cadence, you are not yet ready for a lakehouse.

2.2 The three (four) formats — status verified as of August 2026

What has to be understood about the convergence (it is the fact that decides the choice, and where it is easiest to stay with a 2024 snapshot): the format war ended in a draw, not by extermination. Iceberg won as the interoperability standard; Delta survives as Databricks' native optimised format; and Iceberg's spec v3 absorbed exactly the capabilities that differentiated Delta —deletion vectors, row lineage (_row_id, _last_updated_sequence_number), VARIANT type, column default values, geospatial types, nanosecond timestamps and multi-argument partition transforms—, so the "performance or compatibility" dilemma stopped existing. Databricks has also proposed converging the metadata tree of Iceberg v4 and Delta 5.0 into a single structure: it is a proposal, not a fact — watch it, do not take it as settled (§8).

Practical consequence: the format choice is today a reversible (two-way door) decision and cheap compared with the catalog one. What is lost when changing format is the history: the migration moves metadata and sometimes rewrites files, but the old snapshots and the earlier time travel do not travel. Plan the migration with a coexistence period and do not promise queryable history across the cut.

On Iceberg's format-version: v3 is not backwards compatible — a v2 reader cannot read a v3 table, although a v3 reader reads v2 tables. Raising format-version is a change that breaks old consumers: inventory the engines that read the table before raising it, not after. In v3, positional delete files stop being written in favour of deletion vectors (at most one per data file and snapshot).

2.3 The catalog: this one really is the architecture decision

The format says how the metadata is stored; the catalog says which is the table's current pointer, who can read it and with what credential. It is the serialisation point of the commits, the API boundary between all the engines and your data, and the place where vendor lock-in actually happens. An open format with a closed catalog is not an open lakehouse: it is a product with a public file format.

Catalog rules, in order of importance:

1. Choose an implementation that speaks the Iceberg REST API. It is what lets you change backend without touching each engine's configuration. Starting with Thrift/HMS is signing up for a future migration.
2. Migrating catalog is a metadata operation: the data files do not move, the table is re-registered. That makes the decision less irreversible than it looks — but the real risk is split-brain: two catalogs pointing at the same table during the transition, with two divergent snapshot pointers. Cut writes at the source before registering at the destination, without exception.
3. The catalog is critical stateful infrastructure: its database needs HA, backup and a proven restore (see backup-recovery-standards). If you lose the catalog, the files in the bucket are still there but there is no table. Have the reconstruction procedure from the most recent metadata.json written down.
4. Planned coexistence, not perpetual: it is legitimate to have Glue for the legacy and Polaris for the multi-engine part. Put an end date on the coexistence or it becomes permanent.
5. Mandatory ADR with the exit plan written down: which engines depend on it, how the tables are re-registered and how long it takes.

2.4 Query engines over the same data

What "open" really means: data in Parquet + an open table format is a necessary and not sufficient condition. Check all three, and if one fails do not say it is open: (a) can another engine read the table without exporting it?; (b) can another engine write to it?; (c) does the catalog speak a protocol that someone else implements? The most common scenario of false openness in 2026 is not the format: it is a proprietary catalog that vends credentials only to its own engine, or a managed table where only the vendor can write.

3. Anatomy, partitioning and maintenance

3.1 The metadata tree — understanding it explains 90 % of the performance and cost problems

A table format is a stack of immutable pointers. In Iceberg, top down:

catalog   →  metadata.json            (schema, partitioning, properties, snapshot list)
             └─ snapshot              (a complete state of the table at an instant)
                └─ manifest list      (that snapshot's manifests, with partition ranges)
                   └─ manifest file   (the data files, with per-column metrics: min/max/nulls)
                      └─ data files   (.parquet) + delete files / deletion vectors

Delta uses a transaction log (_delta_log/ with incremental JSON and periodic checkpoints) and Hudi a timeline (.hoodie/), but the three consequences are the same:

• A write modifies nothing: it adds. A commit is publishing a new pointer. That is why there is ACID, and that is why the bucket grows by design until somebody cleans up.
• Query planning is done by reading metadata, not by listing the bucket. Pruning happens at two levels: by partition range (in the manifest list) and by per-column min/max statistics (in the manifest). A slow engine is usually reading too many manifests, not too much data.
• The symptoms of "it is slow" and "it costs a lot" are almost always metadata, not compute: too many live snapshots, too many manifests, too many small files, or delete files not merged. Diagnose by counting files and manifests per partition before touching the cluster.
• Per-column metrics cost metadata space: on very wide tables, writing statistics for all 500 columns inflates the manifest and slows down planning. Limit the columns with statistics to the ones that get filtered.

3.2 Partitioning

• Hidden partitioning (Iceberg): the partition is declared as a transformation over a column (days(event_ts), bucket(16, user_id)) and the engine applies it when filtering by the original column. Compared with directory partitioning (Hive style), it eliminates the most frequent class of error in the domain: the query that filters by event_ts but not by the synthetic column dt, and scans the whole table without anyone noticing until the bill arrives. With hidden partitioning that query prunes properly; without it, it does not.
• Partitioning evolution: the specification can be changed without rewriting the history; old data keeps its spec and new data uses the new one. It is one of the best reasons to adopt the format. Two warnings: (a) the engine has to plan over two specs at once, with its cost; (b) it is not an excuse for not thinking about the initial partitioning, it is an emergency exit.
• Classic and vetoed error: partitioning by a high-cardinality column (user_id, order_id, uuid). It generates millions of tiny partitions, sinks planning and multiplies the requests to the bucket. If you need to group by that column, use bucket(N, col) (bounded number of partitions) or ordering/clustering within the partition, not direct partitioning.
• Sizing rule: aim for partitions of hundreds of MB to a few GB. A daily partition producing 3 MB means the right partition was the month.
• Partition by the column you filter by —almost always the event date, not the ingestion one— and verify the pruning by reading the plan, not by assuming it.

3.3 Maintenance — the section nobody plans and the one that decides whether the project survives

The formats bring the primitives; none of them runs by itself. Scheduling them, sizing them and watching them is platform work with real compute cost, and it goes into the budget from day one.

Hard rules:

• Order: compact first, then expire snapshots (so expiration can free the files that compaction made obsolete), then clean orphans and consolidate manifests. Skipping a step weakens the others.
• remove_orphan_files with a wide safety window (3 days by default; never smaller than the maximum duration of an in-flight write). A short window deletes files from an in-flight job and corrupts the table. This is the most dangerous command in the domain: treat it as such.
• Snapshot expiration sets your real time travel and rollback window. Decide it explicitly (typical: 7 days), publish it and do not discover it the day you need to revert.
• Cadence: stream-fed tables, every 1-3 hours; batch tables, daily; orphans and manifests, weekly. Adjust with the metric, not with habit.
• Compaction costs compute and requests: it rewrites data. On large tables it is a budget line of its own and it competes with the queries. Schedule it outside the critical window and limit its concurrency, or it becomes the process that takes the platform down.
• At scale, maintenance is triggered by signal, not by blind cron: number of files, average size, delete-file ratio, number of manifests and snapshot depth per table, with cron as a safety net.
• Never delete files from the bucket "to clean up" nor set an S3 lifecycle rule that expires objects inside the path of a live table. It is the fastest and most definitive way to destroy a table: the catalog will keep pointing at files that no longer exist. Cleanup is done always with the format's primitives (§7). See object-storage-standards for the lifecycle rule, which here is vetoed over active table paths.

3.4 Concurrent writes and optimistic control

• The model is optimistic concurrency: each writer reads the current snapshot, prepares its files and tries to publish the new pointer. If someone else published in the meantime, the commit fails and the writer retries, re-basing its change on the new snapshot, if the conflict allows it.
• What actually happens when two jobs write the same table: two INSERTs to different partitions almost always coexist (it retries and goes through). Two operations that rewrite the same files —two MERGEs, or a MERGE and a compaction over the same partition— genuinely conflict, and one of the two loses its entire work after having spent the compute. At high throughput, the retry turns into livelock: it never manages to publish.
• Rules: one logical writer per table and partition; maintenance does not overlap with the heavy writes on the same partitions; configure retries with backoff and a cap; and if the conflict is chronic, the problem is the partitioning design or the pipeline concurrency, not the number of retries.
• Isolation is at table level. A change that must be atomic across several tables is not, unless the catalog offers it explicitly (it is Nessie's own branching argument). Do not assume multi-table atomicity.
• Branches and tags (Iceberg branches/tags, Nessie): atomic publication, isolated development environments and audit by tag. Useful and correct; each live branch retains snapshots and therefore files: it goes into the maintenance budget.

3.5 Updates and deletes: copy-on-write versus merge-on-read

The change capture mechanism belongs to streaming-cdc-standards; what it does to the table belongs here. It is a per-table decision, written down, not a default inherited from the example you copied.

• MoR without scheduled compaction is a time bomb: the table is fast to write and slower to read every day, with a degradation nobody attributes to the decision that caused it.
• The real cost of DELETE and UPDATE: in CoW, deleting 10 rows spread across 10 files rewrites all 10 files completely. A compliance-mandated deletion affecting one row per partition can rewrite the whole table. Group the deletes into periodic batches instead of applying them row by row.
• Iceberg v3 replaces positional delete files with deletion vectors (bitmap, at most one per data file and snapshot): fewer files, more efficient reading and simpler comparison of deletes between commits. It is a solid reason to move to v3 on MoR tables, once the readers have been inventoried (§2.2).
• Continuous ingestion produces small files by construction. Do not treat it as a defect of the stream: it is the price of latency, and it is paid with compaction. If nobody wants to pay it, the latency you wanted was not necessary.

4. Quality and testing — gates

In increasing order of cost. Those marked as a gate break the build or block publication.

1. Table definition and properties as code, versioned and reviewed: schema, format-version, partition specification, CoW/MoR strategy, compaction and expiration properties. A table created by hand from a console is a table with no owner. CI gate.
2. Every table is registered in the catalog and accessed through the catalog. Path access (s3://.../table/) bypassing the catalog is forbidden: it is the direct route to corruption and to bypassing access control. Review gate.
3. Schema evolution test: the proposed change is applied to a copy and read with the version of every engine that consumes the table. Renaming, reordering and changing type are legal by spec; removing or changing type incompatibly breaks consumers. A format-version change is an incompatible change as long as an old reader exists. CI gate.
4. Published table contract: removing or renaming a consumed column, changing the grain or changing the partition spec breaks the build unless there is registered approval. The published table is an API (consistent with data-warehouse-modeling-standards §4). Gate.
5. Assertions on the data (key uniqueness, non-nulls, referential, ranges): they belong to data-engineering-standards §4 and data-warehouse-modeling-standards §4. Here it is only required that they block snapshot publication, taking advantage of the fact that the format allows writing to a branch and merging only if they pass.
6. Write idempotency test: running the same load twice produces the same table state (same count, same checksum). CI gate on pipelines that write.
7. Concurrency test: two simultaneous writers on the same table in the test environment; it is verified that one retries and neither loses data silently. This test is never done and it is where production failures appear.
8. Table health as a scheduled test, with thresholds (not just a dashboard): number of files per partition, average file size, number of snapshots, number of manifests, delete-file ratio. Exceeding the threshold opens an actionable alert, not an entry in a dashboard nobody looks at.
9. Rollback drill documented and executed on a schedule: reverting a table to an earlier snapshot, measuring how long it takes and checking that the consumers tolerate it. A rollback that has never been tested does not exist.
10. Catalog loss drill: rebuilding a table's registration from its metadata.json in a test environment. It is this domain's specific disaster scenario.
11. Dependencies pinned by hash/digest in every process with warehouse credentials (pyiceberg, delta-rs, connectors, Spark/Trino images). CI gate (§5).

5. Stack security

5.1 Access control: who actually enforces it

The split, verified and with its blind spot:

Derived design rule, and it is the domain's trap: never rely on fine-grained control applied only in the engine if several engines can obtain storage credentials. A Trino row policy is unknown to Spark; and a scoped credential granting read access to the files will hand over the unmasked rows to any client that does not go through the engine with the policy. If you need real RLS/masking: either the catalog enforces it, or you restrict the set of engines that can obtain credentials, or you publish a derived, already-filtered table.

• Credential vending mandatory: users and engines must not have direct bucket credentials. The catalog authenticates and issues temporary least-privilege credentials. A user with S3 keys for the warehouse invalidates the whole catalog's access control.
• The catalog is a high-value target: strong authentication, TLS, no long-lived static credentials, federated identities (see identity-access-management-standards and secrets-management-standards), and retained audit.
• Separation of writing and maintenance: the pipeline identity writes; the maintenance identity deletes. A single role being able to delete data files is the surface with which a table gets destroyed, by mistake or by attack.

5.2 Substrate and supply chain

• Encryption at rest for the bucket and mandatory TLS on access; the detail belongs to object-storage-standards and cryptography-pki-standards.
• Object Lock in compliance mode over live table paths is incompatible with maintenance: expire_snapshots and remove_orphan_files will not be able to delete anything and storage will grow without limit. If there is a regulatory immutability obligation, it is solved with a derived copy in a locked bucket, not by locking the operational table.
• Supply chain — this is not hypothetical: the data ecosystem has been under a sustained campaign since 2025. Verified precedents: Trivy (Mar 2026), LiteLLM on PyPI (1.82.7 and 1.82.8, 24 Mar 2026), Telnyx (Mar 2026), Microsoft's durabletask (1.4.1-1.4.3, May 2026) and elementary-data 0.23.3 (Apr 2026, .pth pattern that runs when the interpreter starts). The current generation, "Mini Shai-Hulud" (CVE-2026-45321), active since the end of Apr 2026, is a worm that propagates between npm and PyPI, extracts OIDC tokens from the memory of the GitHub Actions runner and has managed to forge SLSA level 3 provenance attestations. Mandatory derivations here:
  • Pin by hash/digest everything installed in a process that has warehouse or catalog credentials; no open ranges.
  • Quarantine window of days before adopting a freshly published version in environments with production credentials.
  • Provenance attestation is no longer sufficient proof on its own: combine it with quarantine and with runner least privilege.
  • A table maintenance runner must not have credentials for more than one environment.
• As of Aug 2026 there is no record of a compromise of pyiceberg or deltalake/delta-rs — verify it again before committing to it (§8).

5.3 Personal data and deletion in an immutable format

The technical constraint, which is the only thing this skill decides (the policy belongs to privacy-engineering-standards):

• A DELETE on the table does not delete the data: it creates a new snapshot in which the row does not appear. The data is still in the previous files and is still recoverable by time travel until the snapshots expire and the orphans are cleaned up.
• Therefore, a legally mandated deletion is only complete when: (1) the DELETE is executed; (2) the affected files are compacted or rewritten; (3) all the snapshots that referenced them expire; (4) the orphans are cleaned up; and (5) it is checked that no branch, tag, replica, backup or derived table keeps the row.
• Mandatory design consequence: the snapshot expiration window becomes a compliance parameter, not just an operational one. If the deletion commitment is 30 days, snapshot retention must be shorter, and that has to be stated in writing.
• When the above is not viable (huge history, frequent deletes, a simultaneous immutability obligation), the way out is crypto-shredding or pseudonymisation by design: the decision belongs to privacy-engineering-standards; the structure that makes it possible is decided here and now, because afterwards it is a complete rewrite of the table.
• Minimisation: the PII column that never lands in the table is the one you do not have to delete across an entire history afterwards.

6. Performance and operability

• Diagnose by metadata, not by cluster. Faced with "it is slow", measure in this order: number of data files read, average size, number of manifests, live snapshots, delete-file ratio, and only then CPU and memory. Adding workers to a fragmented table is paying more for the same problem.
• Minimum per-table metrics, with a threshold and an alert: files per partition, average file size, number of snapshots, number of manifests, age of the last compaction, age of the last expiration, total table bytes versus live bytes, and cost of bucket requests (see object-storage-standards).
• Cost: in this model you pay for storage + requests + query compute + maintenance compute. The two line items people forget are the last two. Budget maintenance explicitly; if it does not fit, the table is badly sized or the lakehouse was unnecessary (§2.1).
• Divergence between stored bytes and live bytes: it is the early and direct indicator that expiration or orphan cleanup is not running. Watch it from day one.
• Ordering and clustering within the partition (order by the filter column, Z-order for multidimensional filters) are worth more than adding partitions. They are applied during compaction.
• Bounded column statistics on wide tables: writing them all inflates the manifests.
• Owner per table, published, with its freshness SLA (from `data-engineering-standa

…(truncated)