tangledgroup Apache Parquet 2.2.0
Overview
Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval. It provides high-performance compression and encoding schemes to handle complex nested data in bulk and is supported across many programming languages and analytics tools (Spark, DuckDB, Arrow, Pandas, Polars, Impala, and more).
Parquet was inspired by the Dremel paper's record shredding and assembly algorithm, enabling efficient storage of complex nested structures. The format separates metadata from data, supports per-column compression and encoding, and enables predicate pushdown through column statistics and page indexes.
When to Use
- Reading or writing Parquet files programmatically
- Designing schemas for columnar storage (choosing physical vs logical types)
- Tuning compression and encoding for storage efficiency or query performance
- Implementing Parquet readers or writers
- Understanding Parquet file layout, row groups, column chunks, and pages
- Working with nested data structures in columnar format
- Optimizing analytics workloads that read Parquet (filter pushdown, pruning)
Core Concepts
Hierarchical Structure
A Parquet file is organized hierarchically:
- File — Contains one or more row groups plus file-level metadata.
- Row Group — Horizontal partition of data into rows. Each row group has exactly one column chunk per column. Typical size: 128MB–1GB.
- Column Chunk — Contiguous data for a single column within a row group. Contains one or more pages.
- Page — Indivisible unit of compression and encoding. Types: dictionary pages, data pages (v1/v2), and index pages.
Physical vs Logical Types
Parquet uses minimal physical types for on-disk storage: BOOLEAN, INT32, INT64, INT96 (deprecated), FLOAT, DOUBLE, BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY.
Logical types annotate physical types to express higher-level semantics: STRING, INT(8/16/32/64, signed/unsigned), DECIMAL, DATE, TIME, TIMESTAMP, INTERVAL, JSON, BSON, UUID, FLOAT16, GEOMETRY, GEOGRAPHY, VARIANT. Nested types LIST and MAP use group annotations.
Null Handling via Definition Levels
Nulls in nested structures are encoded using definition levels — an integer per value indicating how deep the non-null path goes. A definition level of 0 means the value is null; the maximum level means fully populated. This avoids storing explicit null markers for every field.
Nested Data via Repetition Levels
Repeated elements (lists, arrays) use repetition levels — an integer per value indicating whether it starts a new repetition (0) or continues the current one (>0). Combined with definition levels, this enables efficient encoding of arbitrarily nested structures.
Advanced Topics
Types and Schema: Physical types, logical types, LIST/MAP encoding, decimal, temporal, geospatial, variant → Types and Schema Encodings: PLAIN, RLE/Bit-Packing, Dictionary, Delta Binary Packed, Delta Length/String, Byte Stream Split → Encodings Compression: SNAPPY, GZIP, ZSTD, LZ4_RAW, BROTLI, LZO and codec selection → Compression Pages and Column Chunks: Data page v1/v2, dictionary pages, column chunk metadata, statistics → Pages and Column Chunks Metadata and File Layout: File structure, parquet.thrift, file footer, schema elements → Metadata and File Layout Optimization Features: Bloom filters, page indexes, size statistics, error recovery → Optimization Features Encryption and Security: AES/GCM and AES/S2E encryption, footer encryption, key tools → Encryption and Security Extensibility and Compatibility: Schema migration, binary protocol extensions, implementation status → Extensibility and Compatibility