vllm-contrib

name	vllm-contrib
description	Contribution guide, model integration, debugging
triggers	["When user wants to contribute to vLLM","When user needs to add a new model","When user wants to debug vLLM issues","When user needs development environment setup"]

vllm-contrib

Overview

Contributing to vLLM involves understanding the codebase structure, adding new models, and following development practices. This skill covers the contribution workflow.

Prerequisites

• Git installed
• Python 3.9-3.12
• CUDA-capable GPU (for testing)
• Basic understanding of transformers and PyTorch

Main Workflow

Step 1: Development Setup

# Fork and clone
git clone https://github.com/YOUR_USERNAME/vllm.git
cd vllm

# Create virtual environment
python -m venv venv
source venv/bin/activate

# Install in development mode
pip install -e ".[dev]"

# Install pre-commit hooks
pre-commit install

Step 2: Code Structure

vllm/
├── model_executor/     # Model implementations
│   ├── models/        # Specific model files
│   └── layers/        # Reusable layers
├── engine/            # LLMEngine, scheduling
├── entrypoints/       # API servers
├── core/              # Core data structures
├── worker/            # GPU worker processes
└── v1/                # V1 engine (experimental)

Step 3: Adding a New Model

Create a new model file in vllm/model_executor/models/:

# vllm/model_executor/models/my_model.py
from typing import Iterable, List, Optional, Tuple

import torch
from torch import nn

from vllm.attention import Attention, AttentionMetadata
from vllm.model_executor.layers.linear import ColumnParallelLinear
from vllm.model_executor.model_loader.weight_utils import default_weight_loader
from vllm.sequence import SamplerOutput

class MyModel(nn.Module):
    def __init__(self, vllm_config, prefix: str = ""):
        super().__init__()
        # Initialize layers

    def forward(
        self,
        input_ids: torch.Tensor,
        positions: torch.Tensor,
        kv_caches: List[torch.Tensor],
        attn_metadata: AttentionMetadata,
    ) -> torch.Tensor:
        # Forward pass
        pass

    def compute_logits(self, hidden_states: torch.Tensor,
                      embedding: nn.Module) -> torch.Tensor:
        # Compute logits
        pass

    def sample(
        self,
        logits: torch.Tensor,
        sampling_metadata,
    ) -> Optional[SamplerOutput]:
        # Sample tokens
        pass

    def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
        # Load weights from checkpoint
        pass

Step 4: Register the Model

Add to vllm/model_executor/models/__init__.py:

_MODEL_REGISTRY = {
    # ... existing models
    "MyModel": ("my_model", "MyModel"),
}

Add to vllm/model_executor/models/registry.py:

_SUPPORTED_MODELS = {
    # ... existing models
    "example-model-name": MyModel,
}

Step 5: Testing

# Run unit tests
pytest tests/models/test_my_model.py -v

# Run integration tests
pytest tests/entrypoints/test_openai_server.py -v

# Test specific model
python -c "from vllm import LLM; llm = LLM('your-model-name')"

Common Patterns

Pattern 1: Debug Mode

# Enable detailed logging
import logging
logging.basicConfig(level=logging.DEBUG)

# Disable CUDA graphs for easier debugging
from vllm import LLM
llm = LLM(model="model", enforce_eager=True)

Pattern 2: Weight Loading Debugging

# Check weight shapes
from vllm.model_executor.model_loader import get_model

model = get_model(model_config)
for name, param in model.named_parameters():
    print(f"{name}: {param.shape}")

# Check loaded weights
def load_weights(self, weights):
    for name, loaded_weight in weights:
        print(f"Loading: {name} -> {loaded_weight.shape}")
        # ... loading logic

Pattern 3: Attention Implementation

from vllm.attention import Attention

self.attn = Attention(
    num_heads=config.num_attention_heads,
    head_size=config.hidden_size // config.num_attention_heads,
    scale=config.scale,
    num_kv_heads=config.num_key_value_heads,
)

Pattern 4: Adding a Custom Layer

# vllm/model_executor/layers/my_layer.py
import torch.nn as nn

class MyCustomLayer(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.linear = nn.Linear(config.hidden_size, config.hidden_size)

    def forward(self, x):
        return self.linear(x)

Troubleshooting

Problem: Import errors in development

Solution:

# Ensure editable install
pip install -e .

# Check Python path
python -c "import vllm; print(vllm.__file__)"

Problem: CUDA extension build fails

Solution:

# Clean build artifacts
rm -rf build/
rm -rf vllm/*.so
rm -rf ~/.cache/torch_extensions/

# Rebuild
pip install -e . --no-build-isolation -v

Problem: Tests fail with CUDA errors

Solution:

# Run with single GPU
CUDA_VISIBLE_DEVICES=0 pytest tests/

# Run CPU-only tests
pytest tests/ -m "not cuda"

Problem: Model outputs differ from HuggingFace

Solution:

# Compare outputs step by step
# 1. Check embedding
# 2. Check first layer
# 3. Check attention outputs
# 4. Check final logits

# Use this pattern for debugging
from vllm import LLM
import torch

llm = LLM(model="model", enforce_eager=True)

# Get intermediate outputs
# (requires adding debug prints in model code)

References

• Model Integration - Detailed model integration guide
• Debugging Tips - Common debugging techniques