Comprehensive reference documentation and skill for xFormers, Facebook Research's toolbox to accelerate research on Transformers. Use this skill whenever the user mentions xformers, memory_efficient_attention, FMHA, flash attention, SwiGLU, RMSNorm, RoPE, rope_padded, 2:4 structured sparsity, sparsify24, sequence parallelism, fused all-gather/reduce-scatter, tiled matmul, block-sparse tensors, attention patterns, selective activation checkpointing, forward-backward overlap, tree attention, model parallel linear layers, xformers profiler, Triton kernels for transformers, CUTLASS attention, BlockDiagonalMask, LowerTriangularMask, merge_attentions, or xformers internals and build configuration.
Comprehensive reference documentation and skill for bitsandbytes, the k-bit quantization library for PyTorch enabling accessible large language models. Use this skill whenever the user mentions bitsandbytes, LLM.int8(), QLoRA, 4-bit quantization, 8-bit quantization, NF4, FP4, 8-bit optimizers, block-wise quantization, Int8Params, Params4bit, Linear8bitLt, Linear4bit, StableEmbedding, quantize_blockwise, dequantize_blockwise, quantize_4bit, dequantize_4bit, quantize_nf4, quantize_fp4, GlobalOptimManager, paged optimizers, FSDP integration with quantization, Triton kernels for quantization, CPU/XPU/MPS/HPU backends, CUDA kernels for quantized matmul, or bitsandbytes internals.
NVIDIA Megatron-LM & Megatron Core - GPU-optimized framework for training large language models with tensor parallelism, pipeline parallelism, data parallelism (DDP/FSDP), context parallelism, expert parallelism, FP8/FP4 quantization, CUDA graphs, MoE (Mixture of Experts), multimodal models, and TensorRT-LLM export. Supports GPT, BERT, T5, Mamba, LLaMA, Mixtral, DeepSeek-V3, Qwen3, and custom architectures from 2B to 462B parameters with up to 47% MFU on H100 GPUs.
Comprehensive reference documentation and skill for NVIDIA NCCL (Collective Communications Library), the GPU communication library for multi-GPU and multi-node collectives. Use this skill whenever the user mentions NCCL, all-reduce, all-gather, reduce-scatter, broadcast, gather/scatter, all-to-all, ncclSend/ncclRecv, communicator initialization, CUDA stream group semantics, distributed training communication, NVLink/NVSwitch/InfiniBand/TCP transport behavior, NCCL environment variables, debugging NCCL hangs or performance, NCCL plugins (net/tuner/profiler/env), Device API, GIN, LSA, symmetric memory, user buffer registration, RAS, or NCCL source-code internals.
Comprehensive reference documentation and skill for PyTorch - the GPU-accelerated tensor computation and deep learning framework. Covers tensor operations, automatic differentiation, neural network modules (nn), optimization, distributed training, CUDA support, automatic mixed precision (AMP), torch.compile/Dynamo, TorchScript, FX graph transformation, Inductor backend, ONNX export, quantization, profiling, data loading, probability distributions, FFT, linear algebra, sparse tensors, C++ API (libtorch), operator dispatch, custom operators, and deployment. Based on PyTorch source code analysis.
Comprehensive reference documentation and skill for SGLang - a high-performance serving framework for large language models and multimodal models. Covers SGLang architecture, ServerArgs configuration, OpenAI-compatible API server, native API, offline engine API, attention backends (FlashInfer, FlashAttention, Triton, FlashMLA, cutlass_mla), KV cache management with RadixAttention, paged attention, sampling and decoding (structured outputs, constrained decoding, speculative decoding with EAGLE/ngram/DFlash), distributed inference (tensor/pipeline/expert/data parallelism), PD disaggregation, EPD disaggregation, quantization (FP8, FP4/MXFP4, GPTQ, AWQ, INT4/INT8, Marlin, bitsandbytes, GGUF, modelopt), multi-LoRA batching, multimodal processing (image/audio/video), CUDA graphs, torch.compile, piecewise CUDA graphs, sgl-kernel, sgl-model-gateway (Rust), HiCache, HiSparse, RL/post-training support, checkpoint engine, diffusion models, observability, profiling, supported model architectures (200+ models), hardware p
Comprehensive reference documentation and skill for vLLM - a high-throughput and memory-efficient inference and serving engine for large language models (LLMs). Covers vLLM architecture (V0 and V1), engine APIs (LLMEngine, AsyncLLMEngine, LLM), OpenAI-compatible API server, configuration system, model executor and layers, attention mechanisms (PagedAttention, FlashAttention, MLA), KV cache management, sampling and decoding (beam search, speculative decoding, structured outputs), distributed inference (tensor/pipeline/data/expert parallelism), multimodal processing (image/audio/video), compilation and CUDA graphs, custom kernels, quantization (FP8, GPTQ, AWQ, INT4/INT8, etc.), LoRA and adapters, scheduling and memory management, supported model architectures (200+ models), speculative decoding (EAGLE, Medusa, n-gram), observability and profiling, and hardware platforms (NVIDIA GPU, AMD GPU, CPU, TPU, Intel GPU/XPU).
Comprehensive reference documentation and skill for NVIDIA CUDA C++ - the parallel computing platform and programming model for GPU acceleration. Covers CUDA Programming Guide (Release 13.2) and CUDA C++ Best Practices Guide (Release 13.2). Includes programming model, memory management, asynchronous execution, CUDA graphs, cooperative groups, advanced synchronization, async data copies, TMA unit, L2 cache control, green contexts, virtual memory, IPC, multi-GPU, driver API, math functions, device-callable APIs, compute capabilities, C++ language support, deployment, performance optimization, and CUDA profiler tools (Visual Profiler, nvprof, NVTX).