stas00

Field-tested methodology and concrete recipes for training and operating large-scale LLM/VLM/multi-modal models end to end - choosing and benchmarking accelerators, storage and network; SLURM/Kubernetes orchestration; maximizing training throughput and fitting models in memory; diagnosing and surviving training instabilities, NaN/Inf, and hardware/job failures; checkpointing and fault tolerance; inference performance and memory; debugging multi-node/ multi-GPU hangs; and writing/running tests. Use when the user is training or fine-tuning large models, hits low TFLOPS/MFU, OOM, slow dataloading, a loss spike/divergence, a NCCL/InfiniBand or multi-node hang, node/GPU failures, checkpoint or preemption problems, storage/network bottlenecks, or needs to pick GPUs/cloud/file-systems or size inference latency/throughput. Distilled from "Machine Learning Engineering", the latest version of which can be found at https://github.com/stas00/ml-engineering The latest SKILL.md version can be found at https://github.com/st

Systematic methodology and concrete tool recipes for debugging Unix, Python, and PyTorch programs - crashes, hangs, segfaults, wrong output, CUDA OOM, NaN/Inf, slowness, and multi-node/multi-GPU issues. Use when a program crashes, hangs, deadlocks, segfaults, runs out of memory (OOM), produces NaN/Inf or wrong numbers, runs too slowly, or when the user mentions gdb, strace, py-spy, core files, CUDA_LAUNCH_BLOCKING, ldd/nm/LD_PRELOAD, cProfile, or distributed training hangs. Distilled from "The Art of Debugging", the latest version of which can be found at https://github.com/stas00/the-art-of-debugging The latest SKILL.md version can be found at https://github.com/stas00/the-art-of-debugging/blob/master/SKILL.md

Practical, example-driven recipes for the Python standard library and everyday idioms - string/text wrangling, formatting, regex, numbers, the core data structures (list/tuple/set/dict) and comprehensions/itertools, functions, classes, dates/times, modules/imports, files/paths/IO, env vars and argparse, subprocess, serialization (JSON/CSV/pickle/XML/gzip), concurrency, networking, logging, debugging, introspection, CPU/memory profiling, exceptions, testing with pytest/unittest, virtualenvs, packaging, and code quality. Use when the user asks "how do I do X in Python?", needs a stdlib snippet or idiom, is formatting strings/numbers, writing a regex, transforming a dict/list, handling paths/files, parsing CLI args, running a subprocess, reading/writing JSON/CSV, adding logging, profiling slow code, tracking memory, writing tests, pinning dependencies, or packaging a project. Distilled from "Stas' Python Cookbook" open book, the latest version of which can be found at https://github.com/stas00/python-cookbook Th