awq-quantization

name	awq-quantization
description	Activation-aware weight quantization for 4-bit LLM compression with 3x speedup and minimal accuracy loss. Use when deploying large models (7B-70B) on limited GPU memory, when you need faster inference than GPTQ with better accuracy preservation, or for instruction-tuned and multimodal models. MLSys 2024 Best Paper Award winner.
category	ml-training
version	1.0.0
author	Synthetic Sciences
license	MIT
tags	["Optimization","AWQ","Quantization","4-Bit","Activation-Aware","Memory Optimization","Fast Inference","vLLM Integration","Marlin Kernels"]
dependencies	["autoawq","transformers>=4.45.0","torch>=2.0.0"]

AWQ (Activation-aware Weight Quantization)

4-bit quantization that preserves salient weights based on activation patterns, achieving 3x speedup with minimal accuracy loss.

When to use AWQ

Use AWQ when:

• Need 4-bit quantization with <5% accuracy loss
• Deploying instruction-tuned or chat models (AWQ generalizes better)
• Want ~2.5-3x inference speedup over FP16
• Using vLLM for production serving
• Have Ampere+ GPUs (A100, H100, RTX 40xx) for Marlin kernel support

Use GPTQ instead when:

• Need maximum ecosystem compatibility (more tools support GPTQ)
• Working with ExLlamaV2 backend specifically
• Have older GPUs without Marlin support

Use bitsandbytes instead when:

• Need zero calibration overhead (quantize on-the-fly)
• Want to fine-tune with QLoRA
• Prefer simpler integration

Quick start

Installation

# Default (Triton kernels)
pip install autoawq

# With optimized CUDA kernels + Flash Attention
pip install autoawq[kernels]

# Intel CPU/XPU optimization
pip install autoawq[cpu]

Requirements: Python 3.8+, CUDA 11.8+, Compute Capability 7.5+

Load pre-quantized model

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_name = "TheBloke/Mistral-7B-Instruct-v0.2-AWQ"

model = AutoAWQForCausalLM.from_quantized(
    model_name,
    fuse_layers=True  # Enable fused attention for speed
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Generate
inputs = tokenizer("Explain quantum computing", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Quantize your own model