zarr-python

# Zarr Python ## Overview Zarr is a Python library for storing large N-dimensional arrays with chunking and compression. Apply this skill for efficient parallel I/O, cloud-native workflows, and seamless integration with NumPy, Dask, and Xarray. ## Quick Start ### Installation ```bash uv pip install zarr ``` Requires Python 3.11+. For cloud storage support, install additional packages: ```python uv pip install s3fs # For S3 uv pip install gcsfs # For Google Cloud Storage ``` ### Basic Array Creation ```python import zarr import numpy as np # Create a 2D array with chunking and compression z = zarr.create_array( store="data/my_array.zarr", shape=(10000, 10000), chunks=(1000, 1000), dtype="f4" ) # Write data using NumPy-style indexing z[:, :] = np.random.random((10000, 10000)) # Read data data = z[0:100, 0:100] # Returns NumPy array ``` ## Core Operations ### Creating Arrays Zarr provides multiple convenience functions for array creation: ```python # Create empty array z = zarr.zeros(shape=(10000, 10000), chunks=(1000, 1000), dtype='f4', store='data.zarr') # Create filled arrays z = zarr.ones((5000, 5000), chunks=(500, 500)) z = zarr.full((1000, 1000), fill_value=42, chunks=(100, 100)) # Create from existing data data = np.arange(10000).reshape(100, 100) z = zarr.array(data, chunks=(10, 10), store='data.zarr') # Create like another array z2 = zarr.zeros_like(z) # Matches shape, chunks, dtype of z ``` ### Opening Existing Arrays ```python # Open array (read/write mode by default) z = zarr.open_array('data.zarr', mode='r+') # Read-only mode z = zarr.open_array('data.zarr', mode='r') # The open() function auto-detects arrays vs groups z = zarr.open('data.zarr') # Returns Array or Group ``` ### Reading and Writing Data Zarr arrays support NumPy-like indexing: ```python # Write entire array z[:] = 42 # Write slices z[0, :] = np.arange(100) z[10:20, 50:60] = np.random.random((10, 10)) # Read data (returns NumPy array) data = z[0:100, 0:100] row = z[5, :] # Advanced indexing z.vindex[[0, 5, 10], [2, 8, 15]] # Coordinate indexing z.oindex[0:10, [5, 10, 15]] # Orthogonal indexing z.blocks[0, 0] # Block/chunk indexing ``` ### Resizing and Appending ```python # Resize array z.resize(15000, 15000) # Expands or shrinks dimensions # Append data along an axis z.append(np.random.random((1000, 10000)), axis=0) # Adds rows ``` ## Chunking Strategies Chunking is critical for performance. Choose chunk sizes and shapes based on access patterns. ### Chunk Size Guidelines - **Minimum chunk size**: 1 MB recommended for optimal performance - **Balance**: Larger chunks = fewer metadata operations; smaller chunks = better parallel access - **Memory consideration**: Entire chunks must fit in memory during compression ```python # Configure chunk size (aim for ~1MB per chunk) # For float32 data: 1MB = 262,144 elements = 512×512 array z = zarr.zeros( shape=(10000, 10000), chunks=(512, 512), # ~1MB chunks dtype='f4' ) ``` ### Aligning Chunks with Access Patterns **Critical**: Chunk shape dramatically affects performance based on how data is accessed. ```python # If accessing rows frequently (first dimension) z = zarr.zeros((10000, 10000), chunks=(10, 10000)) # Chunk spans columns # If accessing columns frequently (second dimension) z = zarr.zeros((10000, 10000), chunks=(10000, 10)) # Chunk spans rows # For mixed access patterns (balanced approach) z = zarr.zeros((10000, 10000), chunks=(1000, 1000)) # Square chunks ``` **Performance example**: For a (200, 200, 200) array, reading along the first dimension: - Using chunks (1, 200, 200): ~107ms - Using chunks (200, 200, 1): ~1.65ms (65× faster!) ### Sharding for Large-Scale Storage When arrays have millions of small chunks, use sharding to group chunks into larger storage objects: ```python from zarr.codecs import ShardingCodec, BytesCodec from zarr.codecs.blosc import BloscCodec # Create array with sharding z = zarr.create_array( store='data.zarr', shape=(100000, 100000), chunks=(100, 100), # Small chunks for access shards=(1000, 1000), # Groups 100 chunks per shard dtype='f4' ) ``` **Benefits**: - Reduces file system overhead from millions of small files - Improves cloud storage performance (fewer object requests) - Prevents filesystem block size waste **Important**: Entire shards must fit in memory before writing. ## Compression Zarr applies compression per chunk to reduce storage while maintaining fast access. ### Configuring Compression ```python from zarr.codecs.blosc import BloscCodec from zarr.codecs import GzipCodec, ZstdCodec # Default: Blosc with Zstandard z = zarr.zeros((1000, 1000), chunks=(100, 100)) # Uses default compression # Configure Blosc codec z = zarr.create_array( store='data.zarr', shape=(1000, 1000), chunks=(100, 100), dtype='f4', codecs=[BloscCodec(cname='zstd', clevel=5, shuffle='shuffle')] ) # Available Blosc compressors: 'blosclz', 'lz4', 'lz4hc', 'snappy', 'zlib', 'zstd' # Use Gzip compression z = zarr.create_array( store='data.zarr', shape=(1000, 1000), chunks=(100, 100), dtype='f4', codecs=[GzipCodec(level=6)] ) # Disable compression z = zarr.create_array( store='data.zarr', shape=(1000, 1000), chunks=(100, 100), dtype='f4', codecs=[BytesCodec()] # No compression ) ``` ### Compression Performance Tips - **Blosc** (default): Fast compression/decompression, good for interactive workloads - **Zstandard**: Better compression ratios, slightly slower than LZ4 - **Gzip**: Maximum compression, slower performance - **LZ4**: Fastest compression, lower ratios - **Shuffle**: Enable shuffle filter for better compression on numeric data ```python # Optimal for numeric scientific data codecs=[BloscCodec(cname='zstd', clevel=5, shuffle='shuffle')] # Optimal for speed codecs=[BloscCodec(cname='lz4', clevel=1)] # Optimal for compression ratio codecs=[GzipCodec(level=9)] ``` ## Storage Backends Zarr supports multiple storage backends through a flexible storage interface. ### Local Filesystem (Default) ```python from zarr.storage import LocalStore # Explicit store creation store = LocalStore('data/my_array.zarr') z = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100)) # Or use string path (creates LocalStore automatically) z = zarr.open_array('data/my_array.zarr', mode='w', shape=(1000, 1000), chunks=(100, 100)) ``` ### In-Memory Storage ```python from zarr.storage import MemoryStore # Create in-memory store store = MemoryStore() z = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100)) # Data exists only in memory, not persisted ``` ### ZIP File Storage ```python from zarr.storage import ZipStore # Write to ZIP file store = ZipStore('data.zip', mode='w') z = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100)) z[:] = np.random.random((1000, 1000)) store.close() # IMPORTANT: Must close ZipStore # Read from ZIP file store = ZipStore('data.zip', mode='r') z = zarr.open_array(store=store) data = z[:] store.close() ``` ### Cloud Storage (S3, GCS) ```python import s3fs import zarr # S3 storage s3 = s3fs.S3FileSystem(anon=False) # Use credentials store = s3fs.S3Map(root='my-bucket/path/to/array.zarr', s3=s3) z = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100)) z[:] = data # Google Cloud Storage import gcsfs gcs = gcsfs.GCSFileSystem(project='my-project') store = gcsfs.GCSMap(root='my-bucket/path/to/array.zarr', gcs=gcs) z = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100)) ``` **Cloud Storage Best Practices**: - Use consolidated metadata to reduce latency: `zarr.consolidate_metadata(store)` - Align chunk sizes with cloud object sizing (typically 5-100 MB optimal) - Enable parallel writes using Dask for large-scale data - Consider sharding to reduce number of objects ## Groups and Hierarchies Groups organize multiple arrays hierarchically, similar to directories or HDF5 groups. ### Creating and Using Groups ```python # Create root group root = zarr.group(store='data/hierarchy.zarr') # Create sub-groups temperature = root.create_group('temperature') precipitation = root.create_group('precipitation') # Create arrays within groups temp_array = temperature.create_array( name='t2m', shape=(365, 720, 1440), chunks=(1, 720, 1440), dtype='f4' ) precip_array = precipitation.create_array( name='prcp', shape=(365, 720, 1440), chunks=(1, 720, 1440), dtype='f4' ) # Access using paths array = root['temperature/t2m'] # Visualize hierarchy print(root.tree()) # Output: # / # ├── temperature # │ └── t2m (365, 720, 1440) f4 # └── precipitation # └── prcp (365, 720, 1440) f4 ``` ### H5py-Compatible API Zarr provides an h5py-compatible interface for familiar HDF5 users: ```python # Create group with h5py-style methods root = zarr.group('data.zarr') dataset = root.create_dataset('my_data', shape=(1000, 1000), chunks=(100, 100), dtype='f4') # Access like h5py grp = root.require_group('subgroup') arr = grp.require_dataset('array', shape=(500, 500), chunks=(50, 50), dtype='i4') ``` ## Attributes and Metadata Attach custom metadata to arrays and groups using attributes: ```python # Add attributes to array z = zarr.zeros((1000, 1000), chunks=(100, 100)) z.attrs['description'] = 'Temperature data in Kelvin' z.attrs['units'] = 'K' z.attrs['created'] = '2024-01-15' z.attrs['processing_version'] = 2.1 # Attributes are stored as JSON print(z.attrs['units']) # Output: K # Add attributes to groups root = zarr.group('data.zarr') root.attrs['project'] = 'Climate Analysis' root.attrs['institution'] = 'Research Institute' # Attributes persist with the array/group z2 = zarr.open('data.zarr') print(z2.attrs['description']) ``` **Important**: Attributes must be JSON-serializable (strings, numbers, lists, dicts, booleans, null). ## Integration with NumPy, Dask, and Xarray ### NumPy Integration Zarr arrays implement the NumPy array interface: ```python import numpy as np import zarr z = zarr.zeros((1000, 1000), chunks=(100, 100)) # Use NumPy functions directly result = np.sum(z, axis=0) # NumPy operates on Zarr array mean = np.mean(z[:100, :100]) # Convert to NumPy array numpy_array = z[:] # Loads entire array into memory ``` ### Dask Integration Dask provides lazy, parallel computation on Zarr arrays: ```python import dask.array as da import zarr # Create large Zarr array z = zarr.open('data.zarr', mode='w', shape=(100000, 100000), chunks=(1000, 1000), dtype='f4') # Load as Dask array (lazy, no data loaded) dask_array = da.from_zarr('data.zarr') # Perform computations (parallel, out-of-core) result = dask_array.mean(axis=0).compute() # Parallel computation # Write Dask array to Zarr large_array = da.random.random((100000, 100000), chunks=(1000, 1000)) da.to_zarr(large_array, 'output.zarr') ```