compression-codecs

name	compression-codecs
description	Configure and optimize compression for Zarr arrays. Covers all numcodecs compressors (Blosc, Zstd, LZ4, Gzip, LZMA, BZ2), pre-compression filters (Delta, Quantize, FixedScaleOffset, PackBits), codec pipelines, Blosc thread safety, and the trade-offs between compression speed and ratio.
metadata	{"references":["references/patterns.md","references/examples.md","references/common-issues.md"]}

Compression Codecs for Zarr

Configure, select, and optimize compression codecs for Zarr arrays using numcodecs. This skill covers every compressor and filter in the Zarr ecosystem, thread safety for multi-process workloads, codec pipelines in Zarr v3, and performance trade-offs.

Zarr Performance Guide: https://zarr.readthedocs.io/en/latest/user-guide/performance/ numcodecs Reference: https://numcodecs.readthedocs.io/ GitHub: https://github.com/zarr-developers/numcodecs

Quick Reference: Codec Selection

Codec	Speed (compress)	Speed (decompress)	Ratio	Best For
Blosc+LZ4	Very Fast	Very Fast	Low-Med	Real-time analysis, frequent reads
Blosc+Zstd	Medium	Fast	High	General purpose (v2 default)
Zstd standalone	Medium	Fast	High	Zarr v3 default
Blosc+LZ4HC	Slow	Very Fast	Medium	Write-once, read-many
Gzip	Slow	Medium	Med-High	Interop with non-Python tools
LZ4 standalone	Very Fast	Very Fast	Low	Maximum throughput
LZMA	Very Slow	Very Slow	Very High	Archival only

When to Use This Skill

• Selecting a codec for a new Zarr array or dataset
• Optimizing storage size vs read/write speed
• Configuring pre-compression filters (Delta, Quantize, etc.)
• Debugging compression issues (thread safety, poor ratio, codec not found)
• Migrating between v2 and v3 codec configuration

Core Concepts

1. Blosc: The Meta-Compressor (v2 Default)

Blosc wraps internal algorithms and adds byte-shuffling — the single most impactful setting for numerical data compression. Shuffle rearranges bytes to expose patterns, yielding 10–40× better ratios.

Parameter	Options	Default
cname	blosclz, lz4, lz4hc, snappy, zlib, zstd	blosclz
clevel	0 (none) – 9 (max)	5
shuffle	NOSHUFFLE (0), SHUFFLE (1), BITSHUFFLE (2)	SHUFFLE

from numcodecs import Blosc
Blosc(cname='zstd', clevel=5, shuffle=Blosc.SHUFFLE)    # balanced
Blosc(cname='lz4', clevel=1, shuffle=Blosc.SHUFFLE)     # max speed
Blosc(cname='zstd', clevel=9, shuffle=Blosc.BITSHUFFLE) # max ratio

2. Blosc Thread Safety (CRITICAL)

Blosc's internal threading is not fork-safe. Multi-process use (Dask workers, multiprocessing, joblib) can cause silent data corruption.

from numcodecs import blosc
blosc.use_threads = False  # ALWAYS set this in multi-process environments

# For Dask distributed:
client.run(lambda: setattr(__import__('numcodecs').blosc, 'use_threads', False))

3. Standalone Codecs

Codec	Import	Key Config
Zstd (v3 default)	from numcodecs import Zstd	Zstd(level=3) — levels 1–22
LZ4	from numcodecs import LZ4	LZ4(acceleration=1)
Gzip	from numcodecs import GZip	GZip(level=5) — levels 1–9
Zlib	from numcodecs import Zlib	Zlib(level=4) — levels 1–9
BZ2	from numcodecs import BZ2	BZ2(level=5) — levels 1–9
LZMA	from numcodecs import LZMA	LZMA(preset=6) — presets 0–9

4. Pre-Compression Filters

Filters transform data before compression to improve ratios. Applied in order.

Filter	Use Case	Example
Delta	Monotonic data (timestamps, indices)	Delta(dtype='int64')
Quantize	Reduce float precision	Quantize(digits=3, dtype='float64')
FixedScaleOffset	Convert floats to ints	FixedScaleOffset(offset=273.15, scale=100, dtype='float64', astype='int32')
PackBits	Boolean arrays (8× reduction)	PackBits()
Categorize	String→integer encoding	Categorize(labels=['a','b','c'], dtype='U10', astype='u1')

# v2: filters + compressor
z = zarr.open_array('data.zarr', mode='w', shape=(10000,), dtype='int64', chunks=(1000,),
                    filters=[Delta(dtype='int64')], compressor=Blosc(cname='zstd', clevel=5))

# Chain: Delta → Quantize → compressor
filters=[Delta(dtype='float64'), Quantize(digits=3, dtype='float64')]

5. Zarr v3 Codec Pipeline

v3 replaces compressor + filters with a unified pipeline: array→array → array→bytes → bytes→bytes.

import zarr

# v3 with default Zstd
z = zarr.create_array(store='data.zarr', shape=(1000, 1000), chunks=(100, 100),
                      dtype='float64', zarr_format=3)

# v3 with explicit compressor
z = zarr.create_array(store='data.zarr', shape=(1000, 1000), chunks=(100, 100),
                      dtype='float64', compressors=zarr.codecs.ZstdCodec(level=5))

# No compression
z = zarr.create_array(store='data.zarr', shape=(1000, 1000), chunks=(100, 100),
                      dtype='float64', compressors=None)

6. Codec Selection Decision Tree

Primary constraint?
├── STORAGE SIZE → Zstd level 9 or LZMA (archival only)
├── READ SPEED  → Blosc+LZ4 with SHUFFLE (numerical) or LZ4 standalone
├── WRITE SPEED → LZ4(acceleration=10) or Blosc+LZ4 clevel=1
├── BALANCED    → Blosc+Zstd clevel=3 (v2) or Zstd level=3 (v3)
├── INTEROP     → Gzip (universal) or Zlib (NetCDF compat)
└── DATA TYPE
    ├── Monotonic → Delta filter + any compressor
    ├── Boolean   → PackBits + LZ4
    ├── Integer   → Blosc BITSHUFFLE
    └── Limited precision float → Quantize filter + Zstd

Best Practices

• Start with defaults (Blosc+Zstd for v2, Zstd for v3) — only change if benchmarks justify it
• Always use shuffle for numerical data (SHUFFLE for floats, BITSHUFFLE for ints)
• Never use LZMA/BZ2 for frequently-read data
• Set blosc.use_threads = False in any multi-process environment
• Benchmark on real data — synthetic data gives misleading results
• For cloud storage, prioritize decompression speed over compression ratio
• Compression level has diminishing returns above level 5 for most codecs

References

• Patterns: references/patterns.md — codec configs, filter pipelines, per-variable encoding
• Examples: references/examples.md — benchmarking, real-world codec selection
• Common Issues: references/common-issues.md — corruption, poor ratio, codec errors
• Benchmark Script: assets/codec-comparison.py — runnable codec comparison

External Links

• numcodecs: https://numcodecs.readthedocs.io/
• Blosc: https://www.blosc.org/
• Zstandard: https://facebook.github.io/zstd/
• Squash Benchmark: https://quixdb.github.io/squash-benchmark/