ai-infra-plugins

Comprehensive reference documentation and skill for xFormers, Facebook Research's toolbox to accelerate research on Transformers. Use this skill whenever the user mentions xformers, memory_efficient_attention, FMHA, flash attention, SwiGLU, RMSNorm, RoPE, rope_padded, 2:4 structured sparsity, sparsify24, sequence parallelism, fused all-gather/reduce-scatter, tiled matmul, block-sparse tensors, attention patterns, selective activation checkpointing, forward-backward overlap, tree attention, model parallel linear layers, xformers profiler, Triton kernels for transformers, CUTLASS attention, BlockDiagonalMask, LowerTriangularMask, merge_attentions, or xformers internals and build configuration.

Comprehensive reference documentation and skill for bitsandbytes, the k-bit quantization library for PyTorch enabling accessible large language models. Use this skill whenever the user mentions bitsandbytes, LLM.int8(), QLoRA, 4-bit quantization, 8-bit quantization, NF4, FP4, 8-bit optimizers, block-wise quantization, Int8Params, Params4bit, Linear8bitLt, Linear4bit, StableEmbedding, quantize_blockwise, dequantize_blockwise, quantize_4bit, dequantize_4bit, quantize_nf4, quantize_fp4, GlobalOptimManager, paged optimizers, FSDP integration with quantization, Triton kernels for quantization, CPU/XPU/MPS/HPU backends, CUDA kernels for quantized matmul, or bitsandbytes internals.

NVIDIA Megatron-LM & Megatron Core - GPU-optimized framework for training large language models with tensor parallelism, pipeline parallelism, data parallelism (DDP/FSDP), context parallelism, expert parallelism, FP8/FP4 quantization, CUDA graphs, MoE (Mixture of Experts), multimodal models, and TensorRT-LLM export. Supports GPT, BERT, T5, Mamba, LLaMA, Mixtral, DeepSeek-V3, Qwen3, and custom architectures from 2B to 462B parameters with up to 47% MFU on H100 GPUs.

Comprehensive reference documentation and skill for NVIDIA NCCL (Collective Communications Library), the GPU communication library for multi-GPU and multi-node collectives. Use this skill whenever the user mentions NCCL, all-reduce, all-gather, reduce-scatter, broadcast, gather/scatter, all-to-all, ncclSend/ncclRecv, communicator initialization, CUDA stream group semantics, distributed training communication, NVLink/NVSwitch/InfiniBand/TCP transport behavior, NCCL environment variables, debugging NCCL hangs or performance, NCCL plugins (net/tuner/profiler/env), Device API, GIN, LSA, symmetric memory, user buffer registration, RAS, or NCCL source-code internals.

Comprehensive reference documentation and skill for PyTorch - the GPU-accelerated tensor computation and deep learning framework. Covers tensor operations, automatic differentiation, neural network modules (nn), optimization, distributed training, CUDA support, automatic mixed precision (AMP), torch.compile/Dynamo, TorchScript, FX graph transformation, Inductor backend, ONNX export, quantization, profiling, data loading, probability distributions, FFT, linear algebra, sparse tensors, C++ API (libtorch), operator dispatch, custom operators, and deployment. Based on PyTorch source code analysis.

Comprehensive reference documentation and skill for SGLang - a high-performance serving framework for large language models and multimodal models. Covers SGLang architecture, ServerArgs configuration, OpenAI-compatible API server, native API, offline engine API, attention backends (FlashInfer, FlashAttention, Triton, FlashMLA, cutlass_mla), KV cache management with RadixAttention, paged attention, sampling and decoding (structured outputs, constrained decoding, speculative decoding with EAGLE/ngram/DFlash), distributed inference (tensor/pipeline/expert/data parallelism), PD disaggregation, EPD disaggregation, quantization (FP8, FP4/MXFP4, GPTQ, AWQ, INT4/INT8, Marlin, bitsandbytes, GGUF, modelopt), multi-LoRA batching, multimodal processing (image/audio/video), CUDA graphs, torch.compile, piecewise CUDA graphs, sgl-kernel, sgl-model-gateway (Rust), HiCache, HiSparse, RL/post-training support, checkpoint engine, diffusion models, observability, profiling, supported model architectures (200+ models), hardware p

Comprehensive reference documentation and skill for vLLM - a high-throughput and memory-efficient inference and serving engine for large language models (LLMs). Covers vLLM architecture (V0 and V1), engine APIs (LLMEngine, AsyncLLMEngine, LLM), OpenAI-compatible API server, configuration system, model executor and layers, attention mechanisms (PagedAttention, FlashAttention, MLA), KV cache management, sampling and decoding (beam search, speculative decoding, structured outputs), distributed inference (tensor/pipeline/data/expert parallelism), multimodal processing (image/audio/video), compilation and CUDA graphs, custom kernels, quantization (FP8, GPTQ, AWQ, INT4/INT8, etc.), LoRA and adapters, scheduling and memory management, supported model architectures (200+ models), speculative decoding (EAGLE, Medusa, n-gram), observability and profiling, and hardware platforms (NVIDIA GPU, AMD GPU, CPU, TPU, Intel GPU/XPU).

Comprehensive reference documentation and skill for NVIDIA CUDA C++ - the parallel computing platform and programming model for GPU acceleration. Covers CUDA Programming Guide (Release 13.2) and CUDA C++ Best Practices Guide (Release 13.2). Includes programming model, memory management, asynchronous execution, CUDA graphs, cooperative groups, advanced synchronization, async data copies, TMA unit, L2 cache control, green contexts, virtual memory, IPC, multi-GPU, driver API, math functions, device-callable APIs, compute capabilities, C++ language support, deployment, performance optimization, and CUDA profiler tools (Visual Profiler, nvprof, NVTX).

Comprehensive reference documentation and skill for OpenAI Triton - a language and compiler for writing highly efficient custom Deep-Learning primitives on GPUs. Covers Python API (triton.language, triton.runtime, triton.compiler), MLIR dialects and passes, backends (NVIDIA CUDA, AMD ROCm/HIP), experimental features (Gluon, GSAN, triton_kernels), Proton profiler, tutorials, debugging, and build system.

Comprehensive reference documentation and skill for MLIR (Multi-Level Intermediate Representation) - the extensible compiler infrastructure framework from the LLVM project. Covers the MLIR language reference, IR core concepts (Operations, Values, Blocks, Regions, Types, Attributes), dialect definitions (ODS, TableGen), pass infrastructure, pattern rewriting, dialect conversion, PDLL, all standard dialects (Arith, Math, MemRef, Tensor, Vector, SCF, Affine, Linalg, GPU, SPIR-V, LLVM, Transform, Bufferization, TOSA, Quant, OpenMP, OpenACC, Async, PDL, IRDL, EmitC, etc.), interfaces, traits, Python bindings, C API, tools, and tutorials. Based on MLIR source code analysis.

Comprehensive reference documentation and skill for Ray - a unified framework for scaling AI and Python applications. Covers Ray Core (tasks, actors, objects, scheduling, placement groups, namespaces, runtime environment, fault tolerance, compiled graphs, direct transport), Ray Data (datasets, transformations, datasources, preprocessors, execution engine, streaming), Ray Serve (model serving, deployments, HTTP handling, autoscaling, model composition, multi-app, multiplexing, monitoring, architecture), Ray Train (distributed training with PyTorch, TensorFlow, HuggingFace, XGBoost, LightGBM, Horovod, DeepSpeed, JAX; scaling config, checkpointing, training iterators, collective operations), Ray Tune (hyperparameter tuning, search algorithms, schedulers, analysis, logging, stoppers, trainables, CLI, experiment execution), Ray RLlib (reinforcement learning algorithms, RL modules, learners, environments, connectors, replay buffers, callbacks, multi-agent, offline training, fault tolerance), Ray Cluster (setup, con

Comprehensive reference documentation and skill for TensorFlow - the end-to-end open source platform for machine learning. Covers TensorFlow 2.x Python API (tensors, operations, variables, autograd, tf.function, tf.data, Keras, distributed training), C++ core (graph execution, kernels, session, distributed runtime, grappler), XLA compiler (HLO IR, JIT/AOT compilation, MLIR dialects), TensorFlow Lite (converter, delegates, operators, TFLite Micro), C/C++ APIs, SavedModel format, profiling, debugging, performance optimization, and deployment. Based on TensorFlow 2.x source code analysis.

Comprehensive reference documentation and skill for ONNX Runtime - the cross-platform high-performance inference and training engine for ONNX models. Covers C/C++ API, Python API (InferenceSession, OrtValue, SessionOptions), all Execution Providers (CUDA, TensorRT, OpenVINO, DNNL, CoreML, NNAPI, WebGPU, DirectML, QNN, etc.), graph optimization pipeline, operator kernel system, shape inference, custom operators, training (ORTModule, TrainingSession), quantization, LoRA adapters, IO Binding, language bindings (C#, Java, JavaScript, Rust, Objective-C), WebAssembly deployment, build system (CMake), MLAS acceleration, mobile deployment, profiling, and plugin development. Based on ONNX Runtime source code analysis.

Comprehensive reference documentation and skill for JAX - Google's library for high-performance numerical computing and machine learning research. Covers JAX core (transformations, tracing, jaxprs, pytrees), jax.numpy, jax.lax, jax.nn, jax.random, automatic differentiation (grad, custom_jvp, custom_vjp, checkpoint/remat), automatic vectorization (vmap), parallel computing (pjit, shard_map, Mesh, Sharding), jax.scipy, jax.image, Pallas GPU/TPU kernels, export, FFI, debugging, profiling, distributed computing, experimental features, and JAX Enhancement Proposals (JEPs).

Comprehensive reference documentation and skill for DeepSpeed - the distributed deep learning training and inference optimization library. Covers ZeRO optimization (Stages 0-3), ZeRO-Offload, ZeRO-Infinity, SuperOffload, ZenFlow, pipeline parallelism, tensor parallelism (AutoTP), sequence parallelism (Ulysses/ALST), MoE (Mixture of Experts), inference engines (v1/v2), quantization and compression, custom optimizers (Adam, LAMB, LION, Muon, 1-bit Adam), mixed precision training (FP16/BF16/AMP), activation checkpointing, model checkpointing, communication primitives, launcher, autotuning, elasticity, monitoring, profiling, DeepCompile, DeepNVMe, accelerator abstraction, module injection, and supported model implementations. Based on DeepSpeed source code analysis.

Comprehensive reference documentation and skill for TileLang - a concise domain-specific language (DSL) for developing high-performance GPU/CPU kernels built on Apache TVM. Covers the complete programming model (3-level abstraction), language API (memory management, compute primitives, control flow, data movement), compilation pipeline (TIR lowering, optimization passes, codegen), JIT system, backends (CUDA, ROCm, Metal, CPU, WebGPU), layout system, quantization, autotuning, carver system, debugging tools, and extensive examples (GEMM, FlashAttention, DeepSeek MLA/NSA, sparse/quantized operations). Based on TileLang source code with complete API signatures, parameter tables, and code examples for every function.

Comprehensive reference for XLA (Accelerated Linear Algebra) compiler - covering architecture, operation semantics, HLO IR, compilation pipeline, GPU/CPU/TPU backends, PJRT API, MLIR integration, custom calls, autotuning, SPMD partitioning, debugging tools, and build system.

NVIDIA CUTLASS CUDA Template Library - comprehensive reference for high-performance matrix multiplication (GEMM), convolution, tensor operations, and CuTe DSL across all GPU architectures (Volta through Blackwell)