| name | litdata |
| description | Expert use of the LitData library and work on its codebase. Use when writing or reviewing code that calls litdata (StreamingDataset, StreamingDataLoader, StreamingRawDataset, optimize, map, CombinedStreamingDataset, ParallelStreamingDataset, TokensLoader, serializers, typed media wrappers Text/Audio/Video/Image/Jpeg/File/Tensor/Graph/…, list_media_folder, litdata_collate, train_test_split, complete_dataset, merge_datasets, index_parquet_dataset, index_hf_dataset), answering how-to questions, choosing raw vs optimize vs parquet/HF/MDS, keyed lookup (`key_fn`, `build_keys_index`, `dataset_update`, `get_by_key`), elastic mid-epoch resume, tuning cache/prefetch/shuffle/seed, resolving paths (s3/gs/r2/azure/hf/local:/teamspace via resolver.py), documenting or debugging optimize/map I/O (shared queue, async Downloader download/upload, LITDATA_OPTIMIZE_*), FsProvider vs Downloader, FUSE s3_connections/s3_folders, multi-node DATA_OPTIMIZER_* / num_nodes jobs, POSIX-fast mmap, litsim simulator, or when navigating/editing src/litdata, tests, CI, examples/modality, or debugging streaming / optimize / map. Also use for enable_tracer / Litracer / Perfetto, and multi-worker s3:// FileNotFoundError or obstore-after-fork / Studio R2 data_connection_id Session TypeError. |
LitData
LitData (import litdata) preprocesses and streams datasets for PyTorch training:
- Write (
optimize / map) → chunked chunk-*.bin + index.json → src/litdata/processing/
- Read (
StreamingDataset + StreamingDataLoader) → cache → decode → batch → src/litdata/streaming/
- Raw (
StreamingRawDataset) → stream original files without optimize → src/litdata/raw/
To use the library expertly: always load reference/using-litdata.md first. Narrative source: repo README.md.
Install this skill
From any project (Cursor, Claude Code, and other agents supported by the skills CLI):
npx skills add Lightning-AI/litData
Useful options: -g (user-global), -a cursor (Cursor only), -y (non-interactive). In this repo the skill already lives at .claude/skills/litdata/.
Expert usage (load using-litdata.md)
Before writing examples or answering how-tos, read the cookbook. Highlights:
| Topic | Remember |
|---|
| Raw files | StreamingRawDataset + torch DataLoader — #stream-raw / §10. Prefer cloud URL / connection path over FUSE. Defaults: max_concurrent_downloads=None (adaptive Stage 1), max_prefetch=16 (worker-aware ~64 aggregate), hedge_delay=0, download_timeout=120 (batch-level), range_parallel_threshold=0. Explicit int concurrency = exact permits. |
| Images / media | Always wrap so a caption is not a path: Text / Image / Audio / Video / File / … (types.py). Class folders: list_media_folder(root, kind=…). Recipes: examples/modality/. README #modality (#media-types). Bare PIL.Image / fromarray → huge PIL RAW. litdata_collate only special-cases graphs — Audio/Video need a custom collate_fn. |
| Train stream | Optimized: StreamingDataLoader + shuffle=True, drop_last=True, seed=… |
| Optimize | if __name__ == "__main__"; exactly one of chunk_bytes | chunk_size. Default 64MB; multi‑MB samples → consider 256–512MB. Shuffle the sample list before optimize() when source order matters — README #faq-chunk-shuffle |
| Ordered data | Chunk/item shuffle ≠ file-level shuffle. Shuffle before optimize, or use StreamingRawDataset + DataLoader(shuffle=True). LitData does distributed + within-chunk bucket sampling automatically |
| Cache | Peak disk ≈ num_workers × max_pre_download × chunk_size; default max_cache_size=None (75% of free disk, leave ≥50GB). "100G" or 0.90 of free. |
Reference map
| Task | Read |
|---|
| Use the library (raw, optimize/stream, modality, parquet/HF, serializers) | reference/using-litdata.md |
Paths / URLs / Studio mounts / Dir / time templates | reference/resolver.md (+ README #resolve-paths) |
| Downloaders / uploaders / removers / FUSE→cloud / optimize I/O | reference/data-movement.md (+ processing.md) |
Multi-node optimize/map (num_nodes, DATA_OPTIMIZER_*, index merge) | reference/multi-node.md (+ processing.md) |
| Read path, shuffle math, item loaders, Combined/Parallel | reference/streaming.md |
Cache / BinaryWriter / BinaryReader / index.json / FsProvider / sampler | reference/storage-format.md |
| Cache / prefetch / eviction / shared-chunk deletion | reference/cache-and-chunk-lifecycle.md |
Env vars (async prefetch, cache, debug, DATA_OPTIMIZER_*, Studio) | reference/env-vars.md |
Fair streaming benchmarks (benchmarks/ suite) | reference/benchmarking.md |
Offline streaming what-if (litsim) | repo simulator/README.md |
| Raw adaptive concurrency / look-ahead stages (clients own rate) | repo benchmarks/ADAPTIVE_CONCURRENCY.md |
| Lightning Studio env, credentials, free-threading | reference/lightning-studio.md |
| Write path orchestration; raw internals; pointers to I/O + multi-node | reference/processing.md |
| Dev env, PR/CI style | reference/contributing.md |
| Tests & fixtures | reference/testing.md |
Public API (src/litdata/__init__.py)
| Symbol | Purpose |
|---|
StreamingDataset / StreamingDataLoader | Optimized stream + resumable loader |
CombinedStreamingDataset / ParallelStreamingDataset | Mix or zip streams |
StreamingRawDataset | Raw file stream |
TokensLoader | Token windows for LLMs |
optimize / map / merge_datasets / walk / complete_dataset | Write / transform / merge / finish index |
list_media_folder / iter_webdataset_tar | Class-folder / WebDataset ingest lists |
Audio Video Image Jpeg JpegArray Pil Tiff File Mesh Pdf Nifti Tensor Text Graph | Typed media leaves for optimize |
litdata_collate | Loader default; PyG Batch only for graphs |
dataset_update / build_keys_index | Keyed in-place patch / backfill sidecar |
train_test_split | Split by chunk ROIs |
index_parquet_dataset / index_hf_dataset | Index for streaming |
breakpoint | Multiprocessing-safe pdb |
enable_tracer (litdata.debugger) | Pipeline log → Litracer / Perfetto |
Defined under streaming/, processing/, raw/, utilities/ — see cookbook §6–9 for constructor args.
Package map
streaming/ — read · processing/ — write · raw/ — raw stream · types.py — media wrappers · cli/ — litdata cache path|clearutilities/ — env, encryption, subsample, split, parquet, HFconstants.py — optional-dep flags, env knobs, default chunk 64 MB- Registries: downloaders, fs providers, serializers, compressors;
resolver.py → Dir
Shared concepts
- Chunk:
[num_items][offsets][data]; index.json holds chunks + config (data_format, item_loader, …) — storage-format.md.
- Item loaders own layout + intervals (
PyTreeLoader, TokensLoader, ParquetLoader).
- Training downloads =
Downloader. Optimize remote I/O prefers the same Downloader (adownload_file / aupload_file); FsProvider remains for index merge and fallback. See data-movement.md. Sampler ChunkedIndex is read-path; CacheBatchSampler is CacheDataLoader only.
- Ranks from env (
_DistributedEnv / DATA_OPTIMIZER_*), not a custom network.
- Shuffle deterministic from
seed+epoch+chunk → resumable (shuffle.py, not sampler.py). Local POSIX-fast keeps FullShuffle unless the mount is Vast/NFS/Lustre/GPFS (or LITDATA_POSIX_FAST=1 → WindowShuffle).
- Design: one less thing to remember; pure PyTorch; backward compatible; test-driven.
Traps (do not “fix” by guessing)
FileNotFoundError / ChunkWaitTimeoutError after ~120s on chunks with num_workers>0 is usually prefetch-thread death or wait timeout — not a missing object. Check stderr + tracer crash.mode="append" vs use_checkpoint=True: append continues chunk indices; checkpoint resumes input slices. Combining them is unsafe; checkpoint filenames / done_till_index are known-fragile (writer.save_checkpoint vs data_processor load).- Azure/HF resolve for read; optimize write providers are
s3/gs/r2 only (_SUPPORTED_PROVIDERS).
dataset_update: local dirs only; ds[int] is positional.- Empty
Tensor() is not tokens. Only 1-D Tensor(array=…) (or path= / bytes= + dtype=) uses the TokensLoader layout. Empty Text() / JpegArray() fail on write.
- Bare
.txt / .npy / .bin path strings pickle the path. Wrap with Text / File / load the array. Bare *.jpg / *.png / *.wav paths are claimed as media.
litdata_collate only batches graphs. A dict with AudioDecoder / video decoder + a stackable field needs a custom collate_fn (see examples/modality/audio.py).
Quick commands
make setup
pre-commit run --all-files
mypy
pytest tests/path/test_x.py::test_name -v --capture=no
litdata cache path
litdata cache clear
Examples: examples/ (modality recipes: examples/modality/). Version: src/litdata/__about__.py.
