Menu
Per-subsystem status report. What is implemented, what is tested, what is planned but unwired, and known limitations.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Run `uv sync` and the four CI gate checks (ruff check, ruff format, pyright, pytest unit). First command after cloning. Auto-handles non-CUDA hosts via `--no-sources`.
Add a new optimizer to KempnerForge. Covers implementation, registry hook, config fields, tests, and a preset TOML.
First-run setup. Detect SLURM account/partition/QoS from env and write configs/cluster/local.toml so every other skill can preflight.
Walk through KempnerForge's subsystems in the order a forward pass encounters them. Starting point for anyone new to the codebase.
Submit a training job via sbatch. Wraps singlenode.sh for one node and multinode.sh for multiple. Injects account, partition, QoS, and time overrides from local.toml.
End-to-end one-GPU sanity check. Runs a short training loop to confirm torch, CUDA, NCCL, uv, and the dataloader all work before committing to longer runs.
| name | component-gaps |
| description | Per-subsystem status report. What is implemented, what is tested, what is planned but unwired, and known limitations. |
Complements /kempnerforge:explain-architecture (which says what each subsystem does) by adding what each subsystem does not yet do.
Run:
uv run python scripts/check_env.py
Baseline only. If non-zero, fix the baseline before the file references below will resolve.
Subsystems surveyed: model, distributed (FSDP/TP/PP/EP/CP), data, training, checkpoint, resilience, metrics, profiling Unit test coverage (per suite): tests/unit/test_{model, moe, router, optimizer, additional_optimizers, scheduler_extensions, data, packing, mixing, annealing, loss, checkpoint, resilience, distributed, pipeline_parallel, fp8, hooks, training, training_hooks, eval, generate, observability, performance, config}.py Distributed tests: tests/distributed/ (multi-GPU via torchrun) E2E tests: tests/e2e/ (opt-in via --e2e flag) Integration tests: tests/integration/ (single GPU) Known stubs: context parallelism (kempnerforge/distributed/* — no CP module yet)
Assume preflight has passed. Walk the user through the status of the subsystems they care about. Use this matrix as the reference truth; verify any specific claim by reading the code before recommending an action.
transformer.py), GQA attention with SDPA + RoPE, SwiGLU MLP, RMSNorm, token embedding with tied or separate output, KV cache path via generate.py, MoE MLP with sigmoid/softmax top-k routing (moe.py, router.py), FP8 mixed precision hooks (hooks.py).train.py is covered by test_moe.py.fully_shard() (parallel.py), tensor parallelism (tensor_parallel.py), pipeline parallelism (pipeline_parallel.py), expert parallelism for MoE (expert_parallel.py), DeviceMesh composition.tests/distributed/ covers FSDP e2e, TP sharding, PP schedule, EP dispatch. tests/e2e/ run real short training on 4 GPUs.context_parallel.py exists. PyTorch 2.11 exposes an experimental ring-attention API; wiring it in is tracked work, not a docs oversight. If a user asks to enable CP, the honest answer is "pending".MemoryMappedDataset (pre-tokenized .npy), HuggingFaceDataset, StreamingHuggingFaceDataset, MixtureDataset with weighted sampling and phase scheduling (annealing), DistributedSampler with set_skip() for mid-epoch resume, StatefulDataLoader.test_data.py, test_packing.py (document-aware packing), test_mixing.py (mixture weights), test_annealing.py (phase transitions).MixtureDataset sub-datasets that are themselves HF streaming (each sub-source must materialize a dataset instance). No shuffling buffer for streaming HF (relies on HF's internal shuffle).optimizer.py); cosine, linear warmup, WSD, none schedulers (scheduler.py); grad clipping, accumulation via maybe_no_sync, NaN detection hooks (grad.py, hooks.py); loss registry (loss.py); eval loop (eval.py).test_optimizer.py, test_additional_optimizers.py, test_scheduler_extensions.py, test_loss.py, test_training.py, test_training_hooks.py, test_eval.py.manager.py), state dict layout including dataloader position (state.py), non-blocking async save (async_save.py), auto-resume via latest symlink.test_checkpoint.py (unit + round-trip), tests/integration/ covers resumption.keep_last_n. No cross-cloud checkpoint (S3/GCS). No partial-shape-compatible resume (e.g., resume a bigger model from a smaller-model checkpoint).signal_handler.py), GPU compute and memory health probes (health.py), NCCL liveness via all-reduce (health.py), elastic rendezvous helpers (elastic.py).test_resilience.py.MetricsTracker with EMA smoothing, WandB and TensorBoard backends (tracker.py), MFU computation (mfu.py), peak memory tracking with optional snapshot export (memory.py), rank-aware logger (logger.py).test_observability.py, test_performance.py.torch.profiler integration (profiler.py), lightweight CUDA timing (cuda_timer.py).Not applicable — this skill is informational. When the user is about to act on a "gap" above, verify it is still a gap before recommending workarounds:
# Confirm CP is still absent
ls kempnerforge/distributed/*context* 2>/dev/null || echo "no CP module yet"
# Confirm an optimizer is still missing
uv run python -c "from kempnerforge.config.registry import registry; import kempnerforge.training.optimizer; print(registry.list('optimizer'))"
benchmarks/ package holds the actual bench scripts (bench_forward.py, bench_moe.py, bench_optimizer.py, mfu_scaling/, moe_expert_parallel/, moe_packed/) — run them directly for scale validation. The MFU benchmarks cover dense up through 70B and MoE up through 32 GPUs with EP./kempnerforge:explain-architecture — the companion "what it does" walkthrough./kempnerforge:add-optimizer — concrete example of extending a subsystem.