| name | sglang |
| description | Comprehensive reference documentation and skill for SGLang - a high-performance serving framework for large language models and multimodal models. Covers SGLang architecture, ServerArgs configuration, OpenAI-compatible API server, native API, offline engine API, attention backends (FlashInfer, FlashAttention, Triton, FlashMLA, cutlass_mla), KV cache management with RadixAttention, paged attention, sampling and decoding (structured outputs, constrained decoding, speculative decoding with EAGLE/ngram/DFlash), distributed inference (tensor/pipeline/expert/data parallelism), PD disaggregation, EPD disaggregation, quantization (FP8, FP4/MXFP4, GPTQ, AWQ, INT4/INT8, Marlin, bitsandbytes, GGUF, modelopt), multi-LoRA batching, multimodal processing (image/audio/video), CUDA graphs, torch.compile, piecewise CUDA graphs, sgl-kernel, sgl-model-gateway (Rust), HiCache, HiSparse, RL/post-training support, checkpoint engine, diffusion models, observability, profiling, supported model architectures (200+ models), hardware platforms (NVIDIA, AMD, Intel, Google TPU, Ascend NPU, Apple Metal, CPU), Python frontend language primitives, diffusion model acceleration, sgl-model-gateway (Rust), reasoning/structured outputs, developer guide, and function calling. Based on SGLang source code analysis.
|
| version | 0.5.9 |
SGLang - High-Performance LLM Serving Framework
Overview
SGLang is a fast serving framework for large language models and vision language models, designed to deliver low-latency and high-throughput inference across a wide range of setups, from a single GPU to large distributed clusters. It powers over 400,000 GPUs worldwide and generates trillions of tokens daily in production.
Key Capabilities:
- Fast serving with RadixAttention for automatic prefix caching
- Zero-overhead CPU scheduler with continuous batching and chunked prefill
- Prefill-decode disaggregation (PD) and encode-prefill-decode disaggregation (EPD)
- Speculative decoding (EAGLE, ngram, DFlash, adaptive)
- Optimized attention kernels (FlashInfer, FlashAttention 3/4, FlashMLA, Triton, cutlass_mla)
- Tensor/pipeline/expert/data parallelism for distributed inference
- Extensive quantization support (FP8, FP4/MXFP4, GPTQ, AWQ, INT4/INT8, Marlin, GGUF, modelopt)
- Multi-LoRA serving with batching support
- Structured outputs (JSON, regex, EBNF) via XGrammar/Outlines/llguidance
- OpenAI-compatible API server with streaming
- 200+ supported model architectures
- Multimodal model support (image, audio, video)
- Diffusion model acceleration (WAN, Qwen-Image, FLUX)
- HiCache for hierarchical KV cache offloading
- RL/post-training backbone with native integrations (AReaL, verl, slime, Tunix)
Supported Hardware: NVIDIA GPUs (GB200/GB300/B300/H100/A100/L40S/Spark/5090), AMD GPUs (MI355/MI300), Intel Xeon CPUs, Google TPUs, Ascend NPUs, Intel XPU, Apple Metal (MPS), Mthreads GPU
SGLang Version: 0.5.9 | Python: 3.10+ | PyTorch: 2.11.0 | License: Apache 2.0
Architecture Overview
+------------------------------------------------------------------+
| Client Applications |
| OpenAI API | Native API (/generate) | gRPC | Offline LLM |
+------------------------------------------------------------------+
| API Server Layer |
| FastAPI/Uvicorn | SSL/TLS | HTTP/2 | Streaming SSE |
| /v1/chat/completions | /v1/completions | /v1/embeddings |
| /generate | /v1/files | /health | /get_model_info |
+------------------------------------------------------------------+
| Tokenizer Manager (ZMQ) |
| Batch Tokenize | Batch Detokenize | Chat Template |
+------------------------------------------------------------------+
| Scheduler (CPU-side) |
| FCFS/Priority Scheduling | RadixAttention Tree Cache |
| Chunked Prefill | Continuous Batching | Memory Management |
| Prefill Delayer | Dynamic Chunking | SWA Eviction |
+------------------------------------------------------------------+
| DP Controller |
| Load Balancer | Request Routing | Cache-Aware Dispatch |
+------------------------------------------------------------------+
| Model Execution (GPU-side) |
| Model Runner | CUDA Graphs | Piecewise CUDA Graphs |
| torch.compile | FlashInfer Batch | Overlap Schedule |
| Two-Batch Overlap | Single-Batch Overlap |
+------------------------------------------------------------------+
| Distributed Computing |
| Tensor Parallel | Pipeline Parallel | Expert Parallel |
| Data Parallel | PD Disaggregation | EPD Disaggregation |
| Elastic EP | EP Load Balancing | Context Parallelism |
+------------------------------------------------------------------+
| Model & Layers |
| Attention (Flash/MLA/NSA) | Linear | MoE | Embedding |
| Rotary Embedding | Layernorm | Activation | Quantization |
| LoRA Layers | Multimodal Encoders | Mamba/Linear Attn |
+------------------------------------------------------------------+
| Memory Management |
| Paged KV Cache | Radix Tree | Token-to-Slot Pool |
| HiCache (Hierarchical) | KV Cache Offloading | GPU Memory |
+------------------------------------------------------------------+
| Kernels & Operators |
| FlashInfer | FlashAttention 3/4 | sgl-kernel (CUDA) |
| DeepGEMM (FP8/FP4) | Triton | CUTLASS | Marlin |
| DeepEP | Aiter (AMD) | cutedsl | Flash RL |
+------------------------------------------------------------------+
| Hardware Backends |
| NVIDIA CUDA | AMD ROCm | Intel XPU | Google TPU/Jax |
| Ascend NPU | Apple MPS | x86 CPU | Mthreads MUSA |
+------------------------------------------------------------------+
Quick Reference
Installation
pip install --upgrade pip && pip install uv && uv pip install sglang
git clone https://github.com/sgl-project/sglang.git && cd sglang
pip install -e "python"
docker run --gpus all --shm-size 32g -p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--ipc=host lmsysorg/sglang:latest \
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 --port 30000
Launch Server
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --port 30000
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-70B-Instruct --tp 8
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \
--enable-lora --lora-paths my_adapter=/path/to/adapter
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \
--disaggregation-mode prefill --port 30000
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \
--speculative-algorithm EAGLE \
--speculative-draft-model-path meta-llama/Llama-3.1-8B-Instruct \
--speculative-num-steps 5 --speculative-eagle-topk 8
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \
--quantization fp8 --kv-cache-dtype fp8_e5m2
Send Requests
import openai
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")
response = client.chat.completions.create(
model="meta-llama/Llama-3.1-8B-Instruct",
messages=[{"role": "user", "content": "Hello!"}],
temperature=0.7,
max_tokens=128,
)
for chunk in client.chat.completions.create(
model="meta-llama/Llama-3.1-8B-Instruct",
messages=[{"role": "user", "content": "Hello!"}],
stream=True,
):
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
import requests
response = requests.post("http://localhost:30000/generate", json={
"text": "The capital of France is",
"sampling_params": {"temperature": 0, "max_new_tokens": 32},
})
Offline Batch Inference
import sglang as sgl
llm = sgl.Engine(model_path="meta-llama/Llama-3.1-8B-Instruct")
outputs = llm.generate(["Hello!", "What is AI?"], {"temperature": 0.7})
for output in outputs:
print(output["text"])
llm.shutdown()
Code Examples
Official examples from the SGLang repository (86 files), organized by category:
| Category | Description |
|---|
examples/runtime/ | Runtime usage: LoRA, multimodal, reward model, hidden states, engine API |
examples/runtime/multimodal/ | Multimodal model examples (VLM) |
examples/runtime/engine/ | Offline engine API usage |
examples/frontend_language/ | SGLang Python frontend: gen, select, control flow |
examples/chat_template/ | Chat templates for tool calling models |
examples/usage/ | ModelOpt quantization, export utilities |
examples/monitoring/ | Prometheus, Grafana, OpenTelemetry setup |
examples/profiler/ | Nsight profiling tools |
examples/checkpoint_engine/ | Checkpoint management |
examples/sagemaker/ | AWS SageMaker deployment |
Benchmark & Utility Scripts
Benchmark Scripts (scripts/benchmark/)
Kernel-level and system-level benchmarks:
| Category | Key Scripts |
|---|
kernels/ | all_reduce, fused_moe, quantization, attention, MoE benchmarks |
hicache/ | HiCache performance: long context, multi-turn, serving |
benchmark_batch/ | Batch processing and tokenizer benchmarks |
lora/ | LoRA serving benchmarks |
mtbench/, mmlu/, gsm8k/ | Standard model quality benchmarks |
prefill_only/ | Embedding and scoring benchmarks |
reasoning_benchmark/ | Reasoning task evaluation |
Utility Scripts (scripts/)
Development and deployment utilities:
killall_sglang.sh - Clean shutdown of SGLang processesconvert_otel_2_perfetto.py - Convert OpenTelemetry traces to Perfetto formatbuild_sgl_deep_gemm.sh - Build DeepGEMM kernelexport_deepseek_nextn.py - Export DeepSeek NextN weights
Reference Documents
| Document | Description |
|---|
| 01 - Overview and Architecture | Project overview, system architecture, core abstractions, data flow, module structure |
| 02 - Installation and Setup | Installation methods, Docker, hardware support, environment setup |
| 03 - Server Configuration | Complete ServerArgs reference, all CLI flags, defaults, and validation |
| 04 - API Reference | OpenAI API, native /generate API, gRPC, offline engine, embeddings, scoring |
| 05 - Supported Models | All supported model architectures, configurations, and usage examples |
| 06 - Attention and KV Cache | Attention backends, RadixAttention, paged attention, KV cache management |
| 07 - Sampling and Decoding | Sampling parameters, structured outputs, constrained decoding, custom logit processors |
| 08 - Distributed Serving | TP, PP, EP, DP, PD disaggregation, EPD, multi-node, context parallelism |
| 09 - Quantization | All quantization methods: FP8, FP4, GPTQ, AWQ, Marlin, GGUF, modelopt, etc. |
| 10 - Speculative Decoding | EAGLE, ngram, DFlash, adaptive speculative decoding configurations |
| 11 - LoRA Adapters | Multi-LoRA serving, loading, batching, eviction policies |
| 12 - Multimodal Support | VLM support, image/video/audio processing, vision encoders |
| 13 - Kernels and Compilation | sgl-kernel, CUDA graphs, torch.compile, piecewise graphs, compilation pipeline |
| 14 - Scheduling and Memory | Scheduler design, memory management, batch scheduling, chunked prefill |
| 15 - Observability and Profiling | Prometheus metrics, OpenTelemetry tracing, profiling tools, debugging |
| 16 - Layers and Operations | Model layers, attention implementations, linear layers, MoE, normalizations |
| 17 - Advanced Features | HiCache, HiSparse, RL support, checkpoint engine, diffusion models, forward hooks |
| 18 - Benchmark and Deployment | Benchmark tools, Docker deployment, Kubernetes, multi-node setup |
| 19 - Hardware Platforms | NVIDIA, AMD, Intel, TPU, Ascend NPU, CPU, MPS platform-specific guides |
| 20 - Environment Variables | Complete environment variable reference with defaults and descriptions |
| 21 - Diffusion Models | Image/video generation with WAN, FLUX, Qwen-Image, quantization, disaggregation |
| 22 - Python Frontend | SGLang frontend language API: gen, select, function, control flow primitives |
| 23 - Model Gateway | Rust-based sgl-model-gateway: load balancing, routing, cache-aware dispatch |
| 24 - Developer Guide | Contributing, adding models, custom kernels, testing, CI/CD, debugging |
| 25 - Reasoning & Structured Outputs | Reasoning parsers, structured outputs, function calling, constrained decoding |
