vllm-quantization

name	vllm-quantization
description	Model quantization (AWQ, GPTQ, FP8, INT8)
triggers	["When user wants to reduce GPU memory usage","When user needs to run larger models on limited hardware","When user wants to use quantized models","When user needs to balance precision vs performance"]

vllm-quantization

Overview

Quantization reduces model precision to decrease memory usage and increase inference speed. vLLM supports multiple quantization methods including AWQ, GPTQ, FP8, and INT8.

Prerequisites

• vLLM 0.5.0+ installed
• Quantized model or calibration dataset (for self-quantization)
• Understanding of precision vs quality trade-offs

Main Workflow

Step 1: AWQ (Activation-aware Weight Quantization)

Features:

• 4-bit quantization
• Best for consumer GPUs
• Good quality preservation

from vllm import LLM

# Load AWQ model
llm = LLM(
    model="TheBloke/Llama-2-7B-AWQ",
    quantization="awq"
)

# Or specify quantization config
llm = LLM(
    model="TheBloke/Llama-2-7B-AWQ",
    quantization="awq",
    dtype="float16"
)

Step 2: GPTQ (General-purpose Post-Training Quantization)

Features:

• 4-bit quantization
• Wide model support
• Flexible group sizes

from vllm import LLM

# Load GPTQ model
llm = LLM(
    model="TheBloke/Llama-2-7B-GPTQ",
    quantization="gptq"
)

# Specify group size if needed
llm = LLM(
    model="TheBloke/Llama-2-7B-GPTQ",
    quantization="gptq",
    dtype="float16"
)

Step 3: FP8 (8-bit Floating Point)

Features:

• Requires Hopper GPUs (H100)
• Minimal quality loss
• Fast inference

from vllm import LLM

# FP8 for KV cache and weights (H100 only)
llm = LLM(
    model="meta-llama/Llama-2-70b-chat-hf",
    quantization="fp8",
    kv_cache_dtype="fp8"
)

Step 4: INT8 (SmoothQuant)

Features:

• 8-bit integer quantization
• Good for older GPUs
• Moderate quality impact

from vllm import LLM

# INT8 quantization
llm = LLM(
    model="model-name",
    quantization="fp8"  # or use smoothquant from neural-compressor
)

Common Patterns

Pattern 1: Memory-constrained Deployment

from vllm import LLM

# Original: 13GB for 7B model
# AWQ: ~4GB for 7B model

llm = LLM(
    model="TheBloke/Llama-2-7B-AWQ",
    quantization="awq",
    gpu_memory_utilization=0.6,  # Can use less GPU memory
    max_model_len=4096
)

Pattern 2: Serving Large Models

from vllm import LLM

# 70B model on 2x A100 with AWQ
llm = LLM(
    model="TheBloke/Llama-2-70B-AWQ",
    quantization="awq",
    tensor_parallel_size=2,
    max_model_len=4096
)

Pattern 3: Quantization Comparison

import time
from vllm import LLM, SamplingParams

models = {
    "fp16": "meta-llama/Llama-2-7b-chat-hf",
    "awq": "TheBloke/Llama-2-7B-AWQ",
    "gptq": "TheBloke/Llama-2-7B-GPTQ"
}

results = {}
for name, model_path in models.items():
    start = time.time()
    llm = LLM(
        model=model_path,
        quantization=name if name != "fp16" else None,
        gpu_memory_utilization=0.9
    )
    load_time = time.time() - start

    # Check memory
    import torch
    memory = torch.cuda.memory_allocated() / 1024**3

    results[name] = {"load_time": load_time, "memory_gb": memory}
    print(f"{name}: {memory:.2f}GB, {load_time:.2f}s")

Pattern 4: Multi-modal with Quantization

from vllm import LLM

# Quantized vision model
llm = LLM(
    model="TheBloke/llava-v1.5-7B-AWQ",
    quantization="awq",
    dtype="float16"
)

Troubleshooting

Problem: Quantized model produces garbage

Solution:

# Ensure correct dtype
llm = LLM(
    model="TheBloke/Llama-2-7B-AWQ",
    quantization="awq",
    dtype="float16"  # AWQ models usually need float16
)

Problem: Out of memory with quantized model

Solution:

# Even quantized models need KV cache memory
llm = LLM(
    model="TheBloke/Llama-2-7B-AWQ",
    quantization="awq",
    max_model_len=2048,  # Reduce context
    gpu_memory_utilization=0.8
)

Problem: Slow inference with GPTQ

Solution:

# Use faster kernels
llm = LLM(
    model="TheBloke/Llama-2-7B-GPTQ",
    quantization="gptq",
    enforce_eager=True  # Sometimes faster for small batches
)

References

• Quantization Methods - Detailed comparison of methods
• Modality Compatibility - Quantization with multimodal models