Empirically tune GGUF model's llama-swap/llama.cpp config for multi-GPU rig — card placement, KV-cache quant, context length, batch sizes, prompt-cache settings. Benchmarks each dial one at a time, verifies with real load. Checks Hugging Face model card for sampling/template. Modes: "bench" (max tok/s, benchmark-maxxing), "agentic" (max context + safe VRAM headroom for coding agents). Use when user wants to tune, optimize, speed up, or fit local LLM (llama.cpp / llama-server / llama-swap / GGUF) on GPUs — "tune a model for speed", "max out context for a model", "what settings for this model", "optimize my llama-swap config", "dial in tok/s".
Empirically tune GGUF model's llama-swap/llama.cpp config for multi-GPU rig — card placement, KV-cache quant, context length, batch sizes, prompt-cache settings. Benchmarks each dial one at a time, verifies with real load. Checks Hugging Face model card for sampling/template. Modes: "bench" (max tok/s, benchmark-maxxing), "agentic" (max context + safe VRAM headroom for coding agents). Use when user wants to tune, optimize, speed up, or fit local LLM (llama.cpp / llama-server / llama-swap / GGUF) on GPUs — "tune a model for speed", "max out context for a model", "what settings for this model", "optimize my llama-swap config", "dial in tok/s".
disable-model-invocation
true
llama-tune
Tune one model's config the way it's actually done: measure, don't guess. Read
the model's architecture, find its recommended settings, then change ONE dial at
a time and benchmark, confirm the winners work together, and verify a real load
fits with headroom before writing the config.
Before anything: don't contend for the GPU
Benchmarks are meaningless if something else is using the cards. Confirm the
rig is idle before any bench or load (llama-server --list-devices should show
cards near-fully free; ask the user if unsure). CPU-only steps (reading
metadata, editing config) are always safe.
Pick the mode
Ask the user which, if not stated:
bench — maximize tok/s for leaderboard/benchmark runs. Smallest card
footprint that fits, context only as large as needed, KV quant chosen purely
for speed. Squeeze.
agentic — maximize usable context for a coding agent, kept reliable.
Largest context the model (native cap) and hardware allow, prompt cache ON,
KV quant that preserves quality, and a VRAM headroom floor so it won't OOM mid
run.
Workflow
1. Resolve the model + read its architecture (CPU only)
Find the GGUF path (from the llama-swap config key, or ask). Then:
Get --vram-mib (per-card TOTAL) and --cards from llama-server --list-devices. This prints arch, weights size, layers, KV heads, native
context, MoE/SSM/SWA flags, KV/token, min cards needed, and rough
max-context floors per (cards, headroom, KV-quant).
Read references/llama-cpp.md "Architecture gotchas" — MoE, hybrid/SSM, and SWA
make the KV estimate a loose upper bound; real footprint is usually much smaller,
so the only trustworthy fit number comes from step 5.
2. Look up the Hugging Face model card
Search/fetch https://huggingface.co/<org>/a model (and the GGUF repo, e.g.
unsloth/bartowski) for recommended inference settings: temperature, top-p,
top-k, min-p, repeat-penalty, chat template / --chat-template-kwargs,
reasoning format, and any "must use these" notes. Use WebFetch. These set the
sampling flags; the skill tunes performance, not quality — keep the card's
sampling unless the user says otherwise.
3. Detect hardware + set the target
From --list-devices: card count and per-card total/free VRAM (don't assume —
rigs differ). For agentic mode, ask the user squeeze vs conservative
headroom and set a floor (e.g. ~1 GiB squeeze, ~2 GiB balanced, ~3 GiB
conservative) that must hold on the tightest card (the main device runs
tighter — see references/llama-cpp.md).
4. Bench each dial — ONE at a time
Establish a baseline (current config), then change a single dial and re-bench.
Use scripts/bench.sh a model <devices> <ts> [flags] (prints clean pp/tg rows).
Dials, in rough priority:
Card count — fewest cards the weights fit on (min from step 1). For
batch=1, fewer cards = faster (pipeline overhead). Bench 3-card vs 2-card vs
1-card as applicable.
KV quant — f16 vs q8_0 (-ctk/-ctv). f16 is often equal/faster and
higher quality when it fits; q8_0 when context needs the space.
ubatch — -ub 512,1024,2048 for prefill (llama-bench accepts a list).
flash-attn, -sm row — only if a model misbehaves or to squeeze more.
Record what helps / hurts / is neutral. Keep gains that are real (outside the
± noise) and free.
5. Combine winners + verify a real load (ground truth)
Benchmarks don't prove a full-context config loads and survives prefill. Take
the winning dials together and run:
scripts/verify_load.sh a model <devices> -ts ... -np 1 -ctk ... -c <ctx> ...
It reports MIN free VRAM across cards, runs a generation, and checks
prompt-cache reuse. Require:
loads + generates coherently,
MIN free ≥ your headroom floor (agentic) — if not, drop context or add a card,
for agentic: req2 cached_tokens > 0 (prompt cache reuse working). If a config
has --no-cache-prompt, test removing it (see references/llama-cpp.md).
For bench mode, also bench at the workload the leaderboard uses (matching
prompt/output token counts) so the reported numbers line up.
6. Apply + re-verify
Edit only the target model's cmd: block in the llama-swap config (see
references/llama-swap.md for structure and house rules), add -np 1, set the
tuned flags, and write a comment summarizing: card count + why, KV choice,
context, before→after tok/s. Then run verify_load.sh once more on the
final block to confirm. Tell the user to restart llama-swap.
Mode decision cheatsheet
dial
bench (max tps)
agentic (max ctx, reliable)
card count
fewest that fit (fastest)
fewest that fit with headroom at full ctx
context
only as large as the bench needs
model native cap, as high as headroom allows
KV quant
whichever decodes faster
f16 if it fits, else q8_0 (preserve quality)
-np
1
1
prompt cache
off is fine (single-shot)
on (--cache-reuse, no --no-cache-prompt)
headroom
minimal
enforce the floor on the tightest card
Scripts
scripts/model_info.py — GGUF metadata + footprint/min-cards/ctx estimate. CPU only. Runs via uv run --with gguf.
scripts/bench.sh — llama-bench wrapper, clean result rows. GPU.
