sglang

name	sglang
description	Comprehensive reference documentation and skill for SGLang - a high-performance serving framework for large language models and multimodal models. Covers SGLang architecture, ServerArgs configuration, OpenAI-compatible API server, native API, offline engine API, attention backends (FlashInfer, FlashAttention, Triton, FlashMLA, cutlass_mla), KV cache management with RadixAttention, paged attention, sampling and decoding (structured outputs, constrained decoding, speculative decoding with EAGLE/ngram/DFlash), distributed inference (tensor/pipeline/expert/data parallelism), PD disaggregation, EPD disaggregation, quantization (FP8, FP4/MXFP4, GPTQ, AWQ, INT4/INT8, Marlin, bitsandbytes, GGUF, modelopt), multi-LoRA batching, multimodal processing (image/audio/video), CUDA graphs, torch.compile, piecewise CUDA graphs, sgl-kernel, sgl-model-gateway (Rust), HiCache, HiSparse, RL/post-training support, checkpoint engine, diffusion models, observability, profiling, supported model architectures (200+ models), hardware platforms (NVIDIA, AMD, Intel, Google TPU, Ascend NPU, Apple Metal, CPU), Python frontend language primitives, diffusion model acceleration, sgl-model-gateway (Rust), reasoning/structured outputs, developer guide, and function calling. Based on SGLang source code analysis.
version	0.5.9

SGLang - High-Performance LLM Serving Framework

Overview

SGLang is a fast serving framework for large language models and vision language models, designed to deliver low-latency and high-throughput inference across a wide range of setups, from a single GPU to large distributed clusters. It powers over 400,000 GPUs worldwide and generates trillions of tokens daily in production.

Key Capabilities:

• Fast serving with RadixAttention for automatic prefix caching
• Zero-overhead CPU scheduler with continuous batching and chunked prefill
• Prefill-decode disaggregation (PD) and encode-prefill-decode disaggregation (EPD)
• Speculative decoding (EAGLE, ngram, DFlash, adaptive)
• Optimized attention kernels (FlashInfer, FlashAttention 3/4, FlashMLA, Triton, cutlass_mla)
• Tensor/pipeline/expert/data parallelism for distributed inference
• Extensive quantization support (FP8, FP4/MXFP4, GPTQ, AWQ, INT4/INT8, Marlin, GGUF, modelopt)
• Multi-LoRA serving with batching support
• Structured outputs (JSON, regex, EBNF) via XGrammar/Outlines/llguidance
• OpenAI-compatible API server with streaming
• 200+ supported model architectures
• Multimodal model support (image, audio, video)
• Diffusion model acceleration (WAN, Qwen-Image, FLUX)
• HiCache for hierarchical KV cache offloading
• RL/post-training backbone with native integrations (AReaL, verl, slime, Tunix)

Supported Hardware: NVIDIA GPUs (GB200/GB300/B300/H100/A100/L40S/Spark/5090), AMD GPUs (MI355/MI300), Intel Xeon CPUs, Google TPUs, Ascend NPUs, Intel XPU, Apple Metal (MPS), Mthreads GPU

SGLang Version: 0.5.9 | Python: 3.10+ | PyTorch: 2.11.0 | License: Apache 2.0

Architecture Overview

+------------------------------------------------------------------+
|                        Client Applications                        |
|  OpenAI API  |  Native API (/generate)  |  gRPC  |  Offline LLM  |
+------------------------------------------------------------------+
|                         API Server Layer                          |
|  FastAPI/Uvicorn  |  SSL/TLS  |  HTTP/2  |  Streaming SSE       |
|  /v1/chat/completions  |  /v1/completions  |  /v1/embeddings     |
|  /generate  |  /v1/files  |  /health  |  /get_model_info        |
+------------------------------------------------------------------+
|                    Tokenizer Manager (ZMQ)                         |
|  Batch Tokenize  |  Batch Detokenize  |  Chat Template           |
+------------------------------------------------------------------+
|                     Scheduler (CPU-side)                           |
|  FCFS/Priority Scheduling  |  RadixAttention Tree Cache          |
|  Chunked Prefill  |  Continuous Batching  |  Memory Management   |
|  Prefill Delayer  |  Dynamic Chunking  |  SWA Eviction           |
+------------------------------------------------------------------+
|                     DP Controller                                  |
|  Load Balancer  |  Request Routing  |  Cache-Aware Dispatch      |
+------------------------------------------------------------------+
|                    Model Execution (GPU-side)                      |
|  Model Runner  |  CUDA Graphs  |  Piecewise CUDA Graphs          |
|  torch.compile  |  FlashInfer Batch  |  Overlap Schedule          |
|  Two-Batch Overlap  |  Single-Batch Overlap                      |
+------------------------------------------------------------------+
|                    Distributed Computing                           |
|  Tensor Parallel  |  Pipeline Parallel  |  Expert Parallel        |
|  Data Parallel  |  PD Disaggregation  |  EPD Disaggregation      |
|  Elastic EP  |  EP Load Balancing  |  Context Parallelism        |
+------------------------------------------------------------------+
|                    Model & Layers                                  |
|  Attention (Flash/MLA/NSA)  |  Linear  |  MoE  |  Embedding       |
|  Rotary Embedding  |  Layernorm  |  Activation  |  Quantization |
|  LoRA Layers  |  Multimodal Encoders  |  Mamba/Linear Attn       |
+------------------------------------------------------------------+
|                    Memory Management                               |
|  Paged KV Cache  |  Radix Tree  |  Token-to-Slot Pool            |
|  HiCache (Hierarchical)  |  KV Cache Offloading  |  GPU Memory   |
+------------------------------------------------------------------+
|                    Kernels & Operators                             |
|  FlashInfer  |  FlashAttention 3/4  |  sgl-kernel (CUDA)         |
|  DeepGEMM (FP8/FP4)  |  Triton  |  CUTLASS  |  Marlin            |
|  DeepEP  |  Aiter (AMD)  |  cutedsl  |  Flash RL                  |
+------------------------------------------------------------------+
|                    Hardware Backends                               |
|  NVIDIA CUDA  |  AMD ROCm  |  Intel XPU  |  Google TPU/Jax       |
|  Ascend NPU  |  Apple MPS  |  x86 CPU  |  Mthreads MUSA          |
+------------------------------------------------------------------+

Quick Reference

Installation

# pip/uv (recommended)
pip install --upgrade pip && pip install uv && uv pip install sglang

# From source
git clone https://github.com/sgl-project/sglang.git && cd sglang
pip install -e "python"

# Docker
docker run --gpus all --shm-size 32g -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --ipc=host lmsysorg/sglang:latest \
    python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \
    --host 0.0.0.0 --port 30000

Launch Server

# Basic server
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --port 30000

# With tensor parallelism
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-70B-Instruct --tp 8

# With LoRA support
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \
    --enable-lora --lora-paths my_adapter=/path/to/adapter

# PD disaggregation
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \
    --disaggregation-mode prefill --port 30000

# With speculative decoding (EAGLE)
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \
    --speculative-algorithm EAGLE \
    --speculative-draft-model-path meta-llama/Llama-3.1-8B-Instruct \
    --speculative-num-steps 5 --speculative-eagle-topk 8

# With quantization (FP8)
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \
    --quantization fp8 --kv-cache-dtype fp8_e5m2

Send Requests

import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

# Chat completions
response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Hello!"}],
    temperature=0.7,
    max_tokens=128,
)

# Streaming
for chunk in client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
):
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

# Native API
import requests
response = requests.post("http://localhost:30000/generate", json={
    "text": "The capital of France is",
    "sampling_params": {"temperature": 0, "max_new_tokens": 32},
})

Offline Batch Inference

import sglang as sgl

llm = sgl.Engine(model_path="meta-llama/Llama-3.1-8B-Instruct")
outputs = llm.generate(["Hello!", "What is AI?"], {"temperature": 0.7})
for output in outputs:
    print(output["text"])
llm.shutdown()

Code Examples

Official examples from the SGLang repository (86 files), organized by category:

Category	Description
examples/runtime/	Runtime usage: LoRA, multimodal, reward model, hidden states, engine API
examples/runtime/multimodal/	Multimodal model examples (VLM)
examples/runtime/engine/	Offline engine API usage
examples/frontend_language/	SGLang Python frontend: gen, select, control flow
examples/chat_template/	Chat templates for tool calling models
examples/usage/	ModelOpt quantization, export utilities
examples/monitoring/	Prometheus, Grafana, OpenTelemetry setup
examples/profiler/	Nsight profiling tools
examples/checkpoint_engine/	Checkpoint management
examples/sagemaker/	AWS SageMaker deployment

Benchmark & Utility Scripts

Benchmark Scripts (scripts/benchmark/)

Kernel-level and system-level benchmarks:

Category	Key Scripts
kernels/	all_reduce, fused_moe, quantization, attention, MoE benchmarks
hicache/	HiCache performance: long context, multi-turn, serving
benchmark_batch/	Batch processing and tokenizer benchmarks
lora/	LoRA serving benchmarks
mtbench/, mmlu/, gsm8k/	Standard model quality benchmarks
prefill_only/	Embedding and scoring benchmarks
reasoning_benchmark/	Reasoning task evaluation

Utility Scripts (scripts/)

Development and deployment utilities:

• killall_sglang.sh - Clean shutdown of SGLang processes
• convert_otel_2_perfetto.py - Convert OpenTelemetry traces to Perfetto format
• build_sgl_deep_gemm.sh - Build DeepGEMM kernel
• export_deepseek_nextn.py - Export DeepSeek NextN weights

Reference Documents

Document	Description
01 - Overview and Architecture	Project overview, system architecture, core abstractions, data flow, module structure
02 - Installation and Setup	Installation methods, Docker, hardware support, environment setup
03 - Server Configuration	Complete ServerArgs reference, all CLI flags, defaults, and validation
04 - API Reference	OpenAI API, native /generate API, gRPC, offline engine, embeddings, scoring
05 - Supported Models	All supported model architectures, configurations, and usage examples
06 - Attention and KV Cache	Attention backends, RadixAttention, paged attention, KV cache management
07 - Sampling and Decoding	Sampling parameters, structured outputs, constrained decoding, custom logit processors
08 - Distributed Serving	TP, PP, EP, DP, PD disaggregation, EPD, multi-node, context parallelism
09 - Quantization	All quantization methods: FP8, FP4, GPTQ, AWQ, Marlin, GGUF, modelopt, etc.
10 - Speculative Decoding	EAGLE, ngram, DFlash, adaptive speculative decoding configurations
11 - LoRA Adapters	Multi-LoRA serving, loading, batching, eviction policies
12 - Multimodal Support	VLM support, image/video/audio processing, vision encoders
13 - Kernels and Compilation	sgl-kernel, CUDA graphs, torch.compile, piecewise graphs, compilation pipeline
14 - Scheduling and Memory	Scheduler design, memory management, batch scheduling, chunked prefill
15 - Observability and Profiling	Prometheus metrics, OpenTelemetry tracing, profiling tools, debugging
16 - Layers and Operations	Model layers, attention implementations, linear layers, MoE, normalizations
17 - Advanced Features	HiCache, HiSparse, RL support, checkpoint engine, diffusion models, forward hooks
18 - Benchmark and Deployment	Benchmark tools, Docker deployment, Kubernetes, multi-node setup
19 - Hardware Platforms	NVIDIA, AMD, Intel, TPU, Ascend NPU, CPU, MPS platform-specific guides
20 - Environment Variables	Complete environment variable reference with defaults and descriptions
21 - Diffusion Models	Image/video generation with WAN, FLUX, Qwen-Image, quantization, disaggregation
22 - Python Frontend	SGLang frontend language API: gen, select, function, control flow primitives
23 - Model Gateway	Rust-based sgl-model-gateway: load balancing, routing, cache-aware dispatch
24 - Developer Guide	Contributing, adding models, custom kernels, testing, CI/CD, debugging
25 - Reasoning & Structured Outputs	Reasoning parsers, structured outputs, function calling, constrained decoding