uw-ssec Zarr Fundamentals
Master Zarr, the cloud-native format for chunked, compressed, N-dimensional arrays. This skill covers the full Zarr-Python 3 API for both v2 and v3 format specifications.
Official Documentation: https://zarr.readthedocs.io/ Zarr Specification: https://zarr-specs.readthedocs.io/ GitHub: https://github.com/zarr-developers/zarr-python
Quick Reference Card
Installation
pixi add zarr numpy numcodecs # pixi (recommended)
pip install zarr[extra] # pip
pip install zarr[remote] # cloud backends
Essential Operations
import zarr
import numpy as np
# Create array (v3 format, default)
z = zarr.create_array(store="data.zarr", shape=(10000, 10000), chunks=(1000, 1000), dtype="float32")
z[:] = np.random.randn(10000, 10000).astype("float32")
# Open existing
z = zarr.open_array("data.zarr", mode="r")
subset = z[0:100, 0:100]
# Groups
root = zarr.open_group("experiment.zarr", mode="w")
obs = root.create_group("observations")
arr = obs.create_array("temperature", shape=(365, 180, 360), chunks=(30, 90, 180), dtype="float32")
# Metadata
root.attrs["project"] = "Climate Study 2025"
arr.attrs["units"] = "Celsius"
print(root.tree())
Decision Tree
Working with chunked N-dimensional arrays? → Zarr is ideal
Need cloud-native storage (S3/GCS/Azure)? → Zarr + cloud backend (see cloud-storage-backends skill)
Need parallel read/write? → Zarr supports concurrent access natively
Need versioning / ACID transactions? → Icechunk with Zarr
Need to reduce object count on cloud? → Zarr v3 sharding
Working with existing v2 data? → zarr-python 3 supports both v2 and v3
Zarr v2 vs v3 Differences
| Feature | v2 | v3 |
|---|---|---|
| Metadata file | .zarray, .zgroup, .zattrs |
zarr.json (single) |
| Default compressor | Blosc | Zstd |
| Sharding | No | Yes (ZEP 0002) |
| Async I/O | No | Yes (native asyncio) |
| Python | 3.10+ | 3.11+ |
| Chunk key format | 0.0.0 (dot) |
c/0/0/0 (path) |
| Codec system | Single compressor + filters | Composable pipeline |
Default to v3 for new projects. Use zarr_format=2 only for backward compatibility.
Core Concepts
1. Array Creation
z = zarr.create_array(store="data.zarr", shape=(5000, 5000), chunks=(500, 500), dtype="float64",
fill_value=0.0, compressors=zarr.codecs.ZstdCodec(level=3))
# Convenience: zarr.zeros(), zarr.ones(), zarr.full(), zarr.empty()
2. I/O Modes
| Mode | Description |
|---|---|
'r' |
Read-only (error if not found) |
'r+' |
Read/write (must exist) |
'w' |
Write (overwrite if exists) |
'w-' |
Create (error if exists) |
'a' |
Append (create or open) |
3. Group Management
Groups provide HDF5-like hierarchical organization. Create with root.create_group("name"), navigate with root["path/to/child"], list with root.keys(), and inspect with root.tree().
4. Metadata and Attributes
Every array and group has .attrs for key-value metadata. Use CF Conventions attributes (units, long_name, standard_name) for scientific data. Use .info for quick summaries and .info_complete() for detailed stats (slow on large arrays).
5. Indexing Modes
| Mode | Syntax | Returns |
|---|---|---|
| Basic slicing | z[0:100, 0:100] |
Contiguous subarray |
| Coordinate | z.vindex[rows, cols] |
Values at (row, col) pairs |
| Mask | z.vindex[bool_mask] |
1D array of True-masked values |
| Orthogonal | z.oindex[rows, cols] |
Cartesian product (rows × cols) |
| Block | z.blocks[0, 0] |
Entire chunk blocks by index |
| Field | z["field_name"] |
Structured dtype field |
→ See references/patterns.md Pattern 4 for detailed indexing examples.
6. Data Types
Zarr supports: all NumPy numeric types (int8–int64, uint8–uint64, float16–float64, complex64/128), bool, fixed-length strings (S10), datetime64, timedelta64, structured/compound dtypes, and variable-length strings (v2 only via VLenUTF8).
7. Thread and Process Safety
- Reads: Thread-safe without synchronization
- Writes: Safe when writing to non-overlapping chunks (no locks needed)
- Overlapping writes: Use
ThreadSynchronizer(v2) orProcessSynchronizer(v2) - Blosc + multiprocessing: Set
blosc.use_threads = Falseto prevent silent corruption - v3 async: Configure with
zarr.config.set({"async.concurrency": 64})
8. Sharding (v3 Only)
Groups multiple chunks into larger shard files to reduce object count on cloud stores.
z = zarr.create_array(store="sharded.zarr", shape=(10000, 10000),
chunks=(2500, 2500), shards=(500, 500), dtype="float32")
- Without sharding: 100 GB @ 1 MB chunks = 100,000 objects
- With sharding (1 GB shards): same data = 100 objects
- Reads still access individual inner chunks via byte-range requests
Best Practices
| Area | Guidance |
|---|---|
| Chunk size | 1–10 MB for cloud, 100 KB–1 MB for local |
| Fill value | Use np.nan for floats, not 0 if 0 is valid |
| Precision | Prefer float32 over float64 when possible (2× savings) |
| Metadata | Always set units, long_name; follow CF Conventions |
| Cloud | Use sharding (v3) + consolidated metadata |
| Concurrency | Write to non-overlapping chunks; set blosc.use_threads = False |
| Safety | Use mode="r" for reads, mode="w-" to prevent overwrites |
References
- Patterns: references/patterns.md — hierarchical stores, remote access, appending, indexing, sharding, concurrency
- Examples: references/examples.md — full working examples with real-world datasets
- Common Issues: references/common-issues.md — v2/v3 confusion, metadata persistence, memory errors, concurrency bugs
- Quickstart Script: assets/zarr-quickstart.py — runnable demo
External Links
- Zarr docs: https://zarr.readthedocs.io/
- Zarr spec: https://zarr-specs.readthedocs.io/
- numcodecs: https://numcodecs.readthedocs.io/
- xarray Zarr I/O: https://docs.xarray.dev/en/stable/user-guide/io.html#zarr
- Cloud-Native Zarr Guide: https://guide.cloudnativegeo.org/zarr/