checkpointing-and-tracking

# Checkpointing and Tracking Use this sub-skill when the task involves Accelerate checkpoint save/load, model export, tracker initialization/logging, distributed-safe logging, profiling, state/RNG handling, or memory cleanup around those workflows. ## Route First - For the core forward/backward/optimizer loop, gradient accumulation, prepared dataloaders, and `skip_first_batches`, use `../training-loop-integration/`. - For FSDP, DeepSpeed, Megatron-LM, or backend-specific checkpoint strategy and consolidation details, use `../distributed-training-backends/`. - For large-model dispatch/offload checkpoint loading, route to the large-model or big-modeling sub-skill if present. ## Fast Workflows - Save and resume same-script training state with `Accelerator.save_state()` and `Accelerator.load_state()`; see `references/checkpointing.md`. - Configure automatic checkpoint folders and retention with `ProjectConfiguration`; see `references/checkpointing.md`. - Register custom checkpointable objects and save/load pre-hooks for extra metadata; see `references/checkpointing.md`. - Initialize trackers with `Accelerator(log_with=...)`, `init_trackers()`, `log()`, `get_tracker()`, and `end_training()`; see `references/tracking-and-logging.md`. - Use `accelerate.logging.get_logger()` for multiprocess-safe logs and ordered rank logs; see `references/tracking-and-logging.md`. - Profile CPU/GPU/XPU sections with `ProfileKwargs` and `accelerator.profile()`; see `references/tracking-and-logging.md`. - Diagnose common failures using `references/troubleshooting.md` before changing training code. ## Bundled Helper - Run `python scripts/checkpoint_tracker_smoke.py --help` for usage. - Run `python scripts/checkpoint_tracker_smoke.py` to verify a CPU-only tiny save/load/custom-tracker/logging smoke test without any external tracker service. ## Key Constraints - `save_state()`/`load_state()` are for resuming the same training script shape, not arbitrary model conversion between unrelated scripts. - Register custom objects only if they implement both `state_dict()` and `load_state_dict()`. - For ordinary model artifact export, prefer `accelerator.unwrap_model()`, `accelerator.get_state_dict()`, `accelerator.save()`, or `accelerator.save_model()` rather than treating a training-state checkpoint as a deployment artifact.