club-3090-llm-serving

# club-3090 LLM Serving > Skill by [ara.so](https://ara.so) — Daily 2026 Skills collection. Community recipes for serving modern LLMs on RTX 3090 (24 GB) hardware. Supports vLLM, llama.cpp, and SGLang engines with validated Docker Compose configs exposing an OpenAI-compatible API on `localhost:8020`. Currently ships Qwen3.6-27B configs for 1× and 2× cards. --- ## Engine Decision Matrix | Need | Engine | Why | |---|---|---| | Max throughput (code/chat) | vLLM dual | 89–127 TPS, MTP n=3, vision, tools | | Full 262K context, no crashes | llama.cpp single | No prefill cliffs, stable tool-use | | 4 concurrent streams @ 262K | vLLM dual turbo | Stream isolation, full feature stack | | Single card, moderate ctx | vLLM default | ~89 TPS, easiest setup | SGLang is currently **blocked** on Qwen3.6-27B — see `models/qwen3.6-27b/sglang/README.md`. --- ## Prerequisites ``` - 1× or 2× NVIDIA RTX 3090 (24 GB each) - Linux (Ubuntu 22.04+ recommended) - Docker + NVIDIA Container Toolkit - NVIDIA driver 580.x+ - ~30 GB free disk per model ``` --- ## Installation & Setup ### 1. Clone the repo ```bash git clone https://github.com/noonghunna/club-3090.git cd club-3090 ``` ### 2. Download and verify a model ```bash # Downloads model weights, verifies SHA, clones Genesis patches bash scripts/setup.sh qwen3.6-27b ``` ### 3. Launch (interactive wizard) ```bash bash scripts/launch.sh # Wizard prompts: engine → card count → workload → boots compose → verifies ``` ### 4. Launch (non-interactive) ```bash # Single card, chat-optimized bash scripts/launch.sh --variant vllm/default # Dual card, 262K context + vision bash scripts/launch.sh --variant vllm/dual # Single card, 262K context, no prefill cliffs bash scripts/launch.sh --variant llamacpp/default # List all available variants bash scripts/switch.sh --list ``` --- ## Key Scripts | Script | Purpose | |---|---| | `scripts/setup.sh <model>` | Preflight checks, model download, SHA verify, Genesis patch clone | | `scripts/launch.sh [--variant X]` | Interactive or direct variant boot; calls switch.sh + verify-full.sh | | `scripts/switch.sh <variant>` | Stateless switcher — tears down old compose, brings up new one | | `scripts/health.sh` | Live health probe: KV %, MTP accept-length, recent TPS, errors | | `scripts/verify.sh` | Quick smoke test (engine-aware via env vars) | | `scripts/verify-full.sh` | 8-check functional test (~1–2 min) | | `scripts/verify-stress.sh` | Boundary stress test: 262K ladder + tool prefill OOM (~5–10 min) | | `scripts/bench.sh` | Canonical TPS benchmark (3 warm + 5 measured runs) | ### Common script usage ```bash # Switch variants without the wizard bash scripts/switch.sh vllm/long-vision bash scripts/switch.sh vllm/dual bash scripts/switch.sh llamacpp/default # Check runtime health bash scripts/health.sh # Output: KV cache %, MTP accept-length rate, recent TPS, error log tail # Run canonical benchmark bash scripts/bench.sh # Runs narrative + code prompts, prints per-run TPS + averages # Full functional verification after a switch bash scripts/verify-full.sh # Stress test (run before relying on long-context) bash scripts/verify-stress.sh ``` --- ## Variant Names Reference ``` vllm/default Single-card, chat-optimized (recommended first start) vllm/dual Dual-card, 262K ctx, vision, tools, MTP n=3 vllm/long-vision Dual-card, long-context + vision workloads vllm/turbo Dual-card, 4 concurrent streams @ 262K llamacpp/default Single-card, full 262K, no prefill cliffs llamacpp/65k Single-card, 65K ctx (faster, more VRAM headroom) llamacpp/dual Dual-card llama.cpp recipe ``` --- ## API Usage (OpenAI-compatible, port 8020) The server exposes a standard OpenAI-compatible API. Use the `openai` Python SDK pointed at `localhost:8020`. ### Python — openai SDK ```python from openai import OpenAI client = OpenAI( base_url="http://localhost:8020/v1", api_key="ignored", # local server, no auth needed ) # Basic chat response = client.chat.completions.create( model="qwen3.6-27b-autoround", messages=[{"role": "user", "content": "Explain KV cache in one paragraph."}], max_tokens=512, ) print(response.choices[0].message.content) ``` ### Python — streaming ```python stream = client.chat.completions.create( model="qwen3.6-27b-autoround", messages=[{"role": "user", "content": "Write a Python quicksort."}], max_tokens=1024, stream=True, ) for chunk in stream: delta = chunk.choices[0].delta.content if delta: print(delta, end="", flush=True) print() ``` ### Python — raw requests (no SDK dependency) ```python import requests, json payload = { "model": "qwen3.6-27b-autoround", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the capital of France?"}, ], "max_tokens": 200, "temperature": 0.7, } resp = requests.post( "http://localhost:8020/v1/chat/completions", headers={"Content-Type": "application/json"}, json=payload, timeout=120, ) resp.raise_for_status() print(resp.json()["choices"][0]["message"]["content"]) ``` ### Python — tool calling ```python tools = [ { "type": "function", "function": { "name": "search_web", "description": "Search the web for recent information", "parameters": { "type": "object", "properties": { "query": {"type": "string", "description": "Search query"}, }, "required": ["query"], }, }, } ] response = client.chat.completions.create( model="qwen3.6-27b-autoround", messages=[{"role": "user", "content": "What's the latest news on CUDA 13?"}], tools=tools, tool_choice="auto", max_tokens=512, ) msg = response.choices[0].message if msg.tool_calls: for call in msg.tool_calls: print(f"Tool: {call.function.name}") print(f"Args: {call.function.arguments}") ``` ### Python — long context (262K, use with llamacpp/default or vllm/dual) ```python # Load a large document with open("large_codebase.txt") as f: document = f.read() response = client.chat.completions.create( model="qwen3.6-27b-autoround", messages=[ {"role": "user", "content": f"Summarize the architecture:\n\n{document}"}, ], max_tokens=1024, ) print(response.choices[0].message.content) ``` ### TypeScript / Node ```typescript import OpenAI from "openai"; const client = new OpenAI({ baseURL: "http://localhost:8020/v1", apiKey: "ignored", }); async function chat(prompt: string): Promise<string> { const response = await client.chat.completions.create({ model: "qwen3.6-27b-autoround", messages: [{ role: "user", content: prompt }], max_tokens: 512, }); return response.choices[0].message.content ?? ""; } // Streaming in Node async function streamChat(prompt: string): Promise<void> { const stream = await client.chat.completions.create({ model: "qwen3.6-27b-autoround", messages: [{ role: "user", content: prompt }], max_tokens: 1024, stream: true, }); for await (const chunk of stream) { process.stdout.write(chunk.choices[0]?.delta?.content ?? ""); } console.log(); } ``` ### curl — quick sanity check ```bash curl -sf http://localhost:8020/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "qwen3.6-27b-autoround", "messages": [{"role": "user", "content": "Capital of France?"}], "max_tokens": 200 }' | jq '.choices[0].message.content' ``` ### curl — list available models ```bash curl -sf http://localhost:8020/v1/models | jq '.data[].id' ``` --- ## Docker Compose Structure Configs live under `models/qwen3.6-27b/vllm/compose/`. Example structure of a single-card compose: ```yaml # models/qwen3.6-27b/vllm/compose/default.yml (representative structure) services: vllm: image: vllm/vllm-openai:v0.20.1rc1.dev16+g7a1eb8ac2 runtime: nvidia environment: - NVIDIA_VISIBLE_DEVICES=0 - CUDA_VISIBLE_DEVICES=0 ports: - "8020:8000" volumes: - ${MODEL_PATH}:/models/qwen3.6-27b - ${PATCH_PATH}:/patches command: > --model /models/qwen3.6-27b --served-model-name qwen3.6-27b-autoround --tensor-parallel-size 1 --max-model-len 65536 --kv-cache-dtype fp8 --speculative-model /models/qwen3.6-27b/mtp_head --num-speculative-tokens 3 --port 8000 ``` For dual-card, `tensor-parallel-size 2` and `NVIDIA_VISIBLE_DEVICES=0,1` are set, and `max-model-len` extends to 262144. --- ## Connecting External Clients ### Open WebUI ``` API Base URL: http://localhost:8020/v1 API Key: (leave blank or type anything) Model: qwen3.6-27b-autoround ``` ### Cline / Cursor / Copilot-compatible tools ```json { "openai.baseURL": "http://localhost:8020/v1", "openai.apiKey": "local", "openai.model": "qwen3.6-27b-autoround" } ``` ### LiteLLM proxy passthrough ```yaml # litellm_config.yaml model_list: - model_name: qwen3.6-27b litellm_params: model: openai/qwen3.6-27b-autoround api_base: http://localhost:8020/v1 api_key: ignored ``` --- ## Repo Layout Quick Reference ``` club-3090/ ├── scripts/ Shared model-aware scripts (setup, launch, bench, health) ├── models/ │ └── qwen3.6-27b/ │ ├── vllm/ │ │ ├── compose/ Docker Compose files (all variants) │ │ └── patches/ tolist_cudagraph, Marlin pad, Genesis pointer │ ├── llama-cpp/ │ │ └── recipes/ Single-card 65K / 262K-max / dual recipes │ └── sglang/ Blocked — watch list only └── docs/ ├── SINGLE_CARD.md 1× 3090 workload → config guide ├── DUAL_CARD.md 2× 3090 workload → config guide ├── HARDWARE.md PCIe vs NVLink, power draw, card compatibility ├── GLOSSARY.md TPS / KV / MTP / TP / prefill cliff definitions ├── CLIFFS.md Prefill cliff root causes and fix landscape ├── COMPARISONS.md Self-host vs cloud cost crossover analysis ├── UPSTREAM.md Tracked upstream issues and PRs └── engines/ Per-engine deep dives (vLLM / llama.cpp / SGLang) ``` --- ## Troubleshooting ### Server won't start — CUDA/driver error ```bash # Check driver version (need 580.x+) nvidia-smi --query-gpu=driver_version --format=csv,noheader # Check NVIDIA Container Toolkit docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi # Check GPU visibility nvidia-smi -L ``` ### Out of VRAM / OOM on prefill ```bash # Check current KV cache usage bash scripts/health.sh # Switch to a config with smaller max-model-len bash scripts/switch.sh llamacpp/65k # 65K ctx, more headroom bash scripts/switch.sh llamacpp/default # 262K but manages prefill correctly ``` ### Prefill cliff (vLLM hangs or errors on large prompts) This is a known DeltaNet architecture issue on Qwen3.6-27B with vLLM. The llama.cpp route avoids it entirely: ```bash bash scripts/switch.sh llamacpp/default # Stress-test it: bash scripts/verify-stress.sh ```