Zarr Python
Overview
Zarr is a Python library for storing large N-dimensional arrays with chunking and compression. Apply this skill for efficient parallel I/O, cloud-native workflows, and seamless integration with NumPy, Dask, and Xarray.
Quick Start
Installation
uv pip install zarr
Requires Python 3.11+ and Zarr v3 (zarr>=3). For cloud storage support, install the matching
fsspec backend:
uv pip install s3fs # For S3
uv pip install gcsfs # For Google Cloud Storage
Basic Array Creation
import zarr
import numpy as np
# Create a 2D array with chunking and compression
z = zarr.create_array(
store="data/my_array.zarr",
shape=(10000, 10000),
chunks=(1000, 1000),
dtype="f4"
)
# Write data using NumPy-style indexing
z[:, :] = np.random.random((10000, 10000))
# Read data
data = z[0:100, 0:100] # Returns NumPy array
Core Workflow
- Create or open an array/group, picking a store appropriate to the environment (local, in-memory, ZIP, S3/GCS).
- Choose chunking aligned to your access pattern (aim for 1-10 MB chunks; rows-first → chunks span columns, and vice versa). This is the single biggest performance lever.
- Pick compression via
compressors= based on workload — Zstandard (the default), Blosc+LZ4 (fast), Gzip (max ratio); compressors=None to disable.
- Read/write with NumPy-style indexing; resize/append as data grows.
- Scale out with Dask (lazy, out-of-core, parallel) or label with Xarray for climate/geospatial data.
- For cloud and many-array stores, consolidate metadata and consider sharding to cut object/file count.
# Minimal end-to-end
import zarr, numpy as np
z = zarr.create_array(store="data/my_array.zarr", shape=(10000, 10000),
chunks=(1000, 1000), dtype="f4")
z[:, :] = np.random.random((10000, 10000))
sub = z[0:100, 0:100] # returns a NumPy array
Routing — where to look
| You need… |
Go to |
| Array create/open, read/write, resize/append, attributes, groups & hierarchies, consolidated metadata |
references/array_operations.md |
| Chunk-size guidelines, aligning chunks to access patterns, sharding, compression codecs & tips |
references/chunking_compression.md |
| Local / in-memory / ZIP / S3 / GCS stores and cloud best practices |
references/storage_backends.md |
| NumPy / Dask / Xarray integration, thread- and process-safe parallel writes |
references/integration.md |
| Performance checklist, profiling, common patterns (time series, large matrices, cloud-native, format conversion), troubleshooting |
references/patterns_performance.md |
| Full API surface |
references/api_reference.md |
Additional Resources
1---2name: alterlab-zarr3description: Chunked, compressed N-dimensional arrays for cloud storage with Zarr — parallel I/O, S3/GCS integration, and NumPy/Dask/Xarray compatibility. Use when storing or reading large N-D scientific arrays, streaming chunked data to/from cloud object stores, or building large-scale scientific computing pipelines. Part of the AlterLab Academic Skills suite.4license: MIT5---67# Zarr Python89## Overview1011Zarr is a Python library for storing large N-dimensional arrays with chunking and compression. Apply this skill for efficient parallel I/O, cloud-native workflows, and seamless integration with NumPy, Dask, and Xarray.1213## Quick Start1415### Installation1617```bash18uv pip install zarr19```2021Requires Python 3.11+ and Zarr v3 (`zarr>=3`). For cloud storage support, install the matching22fsspec backend:23```bash24uv pip install s3fs # For S325uv pip install gcsfs # For Google Cloud Storage26```2728### Basic Array Creation2930```python31import zarr32import numpy as np3334# Create a 2D array with chunking and compression35z = zarr.create_array(36 store="data/my_array.zarr",37 shape=(10000, 10000),38 chunks=(1000, 1000),39 dtype="f4"40)4142# Write data using NumPy-style indexing43z[:, :] = np.random.random((10000, 10000))4445# Read data46data = z[0:100, 0:100] # Returns NumPy array47```4849## Core Workflow50511. **Create or open** an array/group, picking a store appropriate to the environment (local, in-memory, ZIP, S3/GCS).522. **Choose chunking** aligned to your access pattern (aim for 1-10 MB chunks; rows-first → chunks span columns, and vice versa). This is the single biggest performance lever.533. **Pick compression** via `compressors=` based on workload — Zstandard (the default), Blosc+LZ4 (fast), Gzip (max ratio); `compressors=None` to disable.544. **Read/write** with NumPy-style indexing; resize/append as data grows.555. **Scale out** with Dask (lazy, out-of-core, parallel) or label with Xarray for climate/geospatial data.566. **For cloud and many-array stores**, consolidate metadata and consider sharding to cut object/file count.5758```python59# Minimal end-to-end60import zarr, numpy as np61z = zarr.create_array(store="data/my_array.zarr", shape=(10000, 10000),62 chunks=(1000, 1000), dtype="f4")63z[:, :] = np.random.random((10000, 10000))64sub = z[0:100, 0:100] # returns a NumPy array65```6667## Routing — where to look6869| You need… | Go to |70|-----------|-------|71| Array create/open, read/write, resize/append, attributes, groups & hierarchies, consolidated metadata | `references/array_operations.md` |72| Chunk-size guidelines, aligning chunks to access patterns, sharding, compression codecs & tips | `references/chunking_compression.md` |73| Local / in-memory / ZIP / S3 / GCS stores and cloud best practices | `references/storage_backends.md` |74| NumPy / Dask / Xarray integration, thread- and process-safe parallel writes | `references/integration.md` |75| Performance checklist, profiling, common patterns (time series, large matrices, cloud-native, format conversion), troubleshooting | `references/patterns_performance.md` |76| Full API surface | `references/api_reference.md` |7778## Additional Resources7980- **Official Documentation**: https://zarr.readthedocs.io/81- **Zarr Specifications**: https://zarr-specs.readthedocs.io/82- **GitHub Repository**: https://github.com/zarr-developers/zarr-python83- **Related**: Xarray (labeled arrays), Dask (parallel computing), NumCodecs (compression codecs)84
AlterLab-IEU