vlmbench

name

vlmbench

description

VLM benchmark CLI — run, compare, and reproduce VLM inference benchmarks. Use when the user wants to: (1) benchmark a VLM model's throughput, latency, or VRAM usage, (2) compare results across models or configs, (3) choose or configure a backend (vLLM Docker, vLLM native, Ollama, SGLang, cloud API), (4) run text-only LLM benchmarks via HF dataset text columns, (5) look up model-specific --serve-args, or (6) debug server startup or GPU detection issues. Triggers on: vlmbench, VLM benchmarking, OCR model evaluation, inference throughput, tokens/sec, TTFT, TPOT, VRAM, text benchmark.

vlmbench

All CLI logic is in vlmbench/cli.py (single file). Entry point: main().

Running benchmarks

Prefer uvx — no install needed:

# macOS (Ollama auto-starts)
uvx vlmbench run -m qwen3-vl:2b -i ./images/

# Linux (vLLM Docker auto-starts with --gpus all)
uvx vlmbench run -m Qwen/Qwen3-VL-8B-Instruct -i ./images/

# Linux (native vLLM)
uvx vlmbench run -m Qwen/Qwen3-VL-8B-Instruct -i ./images/ --backend vllm

# Remote/cloud server — model auto-detected from GET /v1/models
uvx vlmbench run -i ./images/ --base-url https://my-server.example.com/v1 --api-key $KEY

# HF image dataset
uvx vlmbench run -m Qwen/Qwen3-VL-2B-Instruct \
  -d hf://vlm-run/FineVision-vlmbench-mini --max-samples 64

# HF text dataset (text-only / LLM benchmark)
uvx vlmbench run -m meta-llama/Llama-3.1-8B-Instruct \
  -d hf://my-org/my-prompts --dataset-text-col prompt --prompt ""

# Compare results
uvx vlmbench compare results/*.json

run is the default subcommand — flags like --model implicitly invoke it.

--model is optional

When --base-url points to a running server, -m / --model can be omitted. vlmbench calls GET /v1/models on the server and uses the first returned model ID. The Authorization: Bearer <api-key> header is included automatically when --api-key is set. --model (or --profile) is still required when using --serve to start a new server.

Text-only / LLM benchmarks

Use --dataset-text-col <col> to load a string column from an HF dataset instead of images. Each row's value is sent as a {"type": "text"} content block before the --prompt instruction.

# Column value IS the full message (no instruction)
--dataset-text-col prompt --prompt ""

# Column value is the document; --prompt is the instruction
--dataset-text-col text --prompt "Summarize in one sentence."

Auto-detection: when no image column is found, vlmbench falls back to any column named text, prompt, input, content, query, question, or instruction.

Backend resolution

Platform	auto resolves to	Model ID style
macOS	ollama	qwen3-vl:2b
Linux	vllm-openai:latest (Docker)	Qwen/Qwen3-VL-2B-Instruct

Docker backends use --gpus all --ipc=host. Servers run in tmux sessions (vlmbench-vllm, vlmbench-ollama, vlmbench-sglang).

Model-specific serve-args

See MODELS.md for the full table of tested models and their required --serve-args. Reference this file when the user asks which models are supported or how to serve a specific model.

Concurrency sweep

Use --concurrency 4,8,16,32,64 to run at multiple concurrency levels in a single invocation. Produces one JSON per level (tagged c4, c8, etc.) and a consolidated comparison table. A single value (e.g. --concurrency 8) runs at that level only.

Key metrics

TTFT (ms), TPOT (ms), tok/s, img/s, latency s/img, VRAM peak (MiB), prompt/completion token counts. Results saved as JSON to ./results/{backend}-v{version}-{model}-{gpu}-{tag}.json.

GPU detection

CUDA_VISIBLE_DEVICES is respected via _nvidia_gpu_index() -> nvidia-smi --id={idx}. When VRAM or GPU name looks wrong, check this env var first.