Compression Codecs

Compression Codecs for Zarr

Configure, select, and optimize compression codecs for Zarr arrays using numcodecs. This skill covers every compressor and filter in the Zarr ecosystem, thread safety for multi-process workloads, codec pipelines in Zarr v3, and performance trade-offs.

Zarr Performance Guide: https://zarr.readthedocs.io/en/latest/user-guide/performance/ numcodecs Reference: https://numcodecs.readthedocs.io/ GitHub: https://github.com/zarr-developers/numcodecs

Quick Reference: Codec Selection

When to Use This Skill

• Selecting a codec for a new Zarr array or dataset
• Optimizing storage size vs read/write speed
• Configuring pre-compression filters (Delta, Quantize, etc.)
• Debugging compression issues (thread safety, poor ratio, codec not found)
• Migrating between v2 and v3 codec configuration

Core Concepts

1. Blosc: The Meta-Compressor (v2 Default)

Blosc wraps internal algorithms and adds byte-shuffling — the single most impactful setting for numerical data compression. Shuffle rearranges bytes to expose patterns, yielding 10–40× better ratios.

from numcodecs import Blosc
Blosc(cname='zstd', clevel=5, shuffle=Blosc.SHUFFLE)    # balanced
Blosc(cname='lz4', clevel=1, shuffle=Blosc.SHUFFLE)     # max speed
Blosc(cname='zstd', clevel=9, shuffle=Blosc.BITSHUFFLE) # max ratio

2. Blosc Thread Safety (CRITICAL)

Blosc's internal threading is not fork-safe. Multi-process use (Dask workers, multiprocessing, joblib) can cause silent data corruption.

from numcodecs import blosc
blosc.use_threads = False  # ALWAYS set this in multi-process environments

# For Dask distributed:
client.run(lambda: setattr(__import__('numcodecs').blosc, 'use_threads', False))

3. Standalone Codecs

4. Pre-Compression Filters

Filters transform data before compression to improve ratios. Applied in order.

# v2: filters + compressor
z = zarr.open_array('data.zarr', mode='w', shape=(10000,), dtype='int64', chunks=(1000,),
                    filters=[Delta(dtype='int64')], compressor=Blosc(cname='zstd', clevel=5))

# Chain: Delta → Quantize → compressor
filters=[Delta(dtype='float64'), Quantize(digits=3, dtype='float64')]

5. Zarr v3 Codec Pipeline

v3 replaces compressor + filters with a unified pipeline: array→array → array→bytes → bytes→bytes.

import zarr

# v3 with default Zstd
z = zarr.create_array(store='data.zarr', shape=(1000, 1000), chunks=(100, 100),
                      dtype='float64', zarr_format=3)

# v3 with explicit compressor
z = zarr.create_array(store='data.zarr', shape=(1000, 1000), chunks=(100, 100),
                      dtype='float64', compressors=zarr.codecs.ZstdCodec(level=5))

# No compression
z = zarr.create_array(store='data.zarr', shape=(1000, 1000), chunks=(100, 100),
                      dtype='float64', compressors=None)

6. Codec Selection Decision Tree

Primary constraint?
├── STORAGE SIZE → Zstd level 9 or LZMA (archival only)
├── READ SPEED  → Blosc+LZ4 with SHUFFLE (numerical) or LZ4 standalone
├── WRITE SPEED → LZ4(acceleration=10) or Blosc+LZ4 clevel=1
├── BALANCED    → Blosc+Zstd clevel=3 (v2) or Zstd level=3 (v3)
├── INTEROP     → Gzip (universal) or Zlib (NetCDF compat)
└── DATA TYPE
    ├── Monotonic → Delta filter + any compressor
    ├── Boolean   → PackBits + LZ4
    ├── Integer   → Blosc BITSHUFFLE
    └── Limited precision float → Quantize filter + Zstd

Best Practices

• Start with defaults (Blosc+Zstd for v2, Zstd for v3) — only change if benchmarks justify it
• Always use shuffle for numerical data (SHUFFLE for floats, BITSHUFFLE for ints)
• Never use LZMA/BZ2 for frequently-read data
• Set blosc.use_threads = False in any multi-process environment
• Benchmark on real data — synthetic data gives misleading results
• For cloud storage, prioritize decompression speed over compression ratio
• Compression level has diminishing returns above level 5 for most codecs

References

• Patterns: references/patterns.md — codec configs, filter pipelines, per-variable encoding
• Examples: references/examples.md — benchmarking, real-world codec selection
• Common Issues: references/common-issues.md — corruption, poor ratio, codec errors
• Benchmark Script: assets/codec-comparison.py — runnable codec comparison

External Links

• numcodecs: https://numcodecs.readthedocs.io/
• Blosc: https://www.blosc.org/
• Zstandard: https://facebook.github.io/zstd/
• Squash Benchmark: https://quixdb.github.io/squash-benchmark/