ollama37-ci-perf

name	ollama37-ci-perf
description	Run a decision-grade FA/CUDA performance experiment via CI — verified throughput, self-reverting testbed
user-invocable	true

Run an ollama37 performance experiment with CI

Measure a CUDA / flash-attention change's real performance through test-throughput.yml on a self-hosted runner — tok/s that can actually decide whether to keep or drop the change. Build the variant under test with ollama37-ci-build; apply an image with ollama37-ci-apply.

$ARGUMENTS

Decision-grade, or it doesn't count

Raw tok/s from a hand curl loop is not evidence for an FA decision — it is single-run and never checks the answer is correct; a path that is faster but emits garbage is not a win. Decision-grade means all three:

• Verified response — judge_mode=dual runs the keyless agent judge for meaningfulness, not just "non-empty." Required for any keep/drop decision. dual silently degrades to simple if the runner has no agent-judge auth (~/.claude) — so confirm a judge verdict actually appears in the run; if it fell back to simple, the response is not verified and the number is not decision-grade.
• Realistic context — context_size ≥ ~6.8k, not a one-line prompt. A short prompt once hid a 7.4× flash-attention regression on the K80 behind a 22% one (docs/porting-k80.md §3).
• Consistent — the workflow recreates the container per run, so the state is fixed; compare paths on the same model + context.

A "path" is (build) × (runtime env)

• --ref <branch> selects the kernel routing that got compiled — FA-off, all-VEC, decode-only, MMA-on, … — i.e. ollama37-ci-build's output.
• flash_attention selects FA vs the non-FA cuBLAS path at runtime: 0 = cuBLAS baseline, 1 = FA. kv_cache_type is the other runtime knob.

Run one

gh workflow run test-throughput.yml --ref <branch> \
  -f runner_label=sm75 \
  -f models=<model> \
  -f flash_attention=<0|1> \
  -f kv_cache_type='' \
  -f context_size=8192 -f num_predict=128 \
  -f judge_mode=dual
gh run watch <run-id> --exit-status

Read the markdown table in the run summary: Prompt tok/s = prefill, Gen tok/s = decode, Check = PASS/FAIL (the judge's verdict). Collect one row per path and compare.

The testbed reverts itself — fail is data

test-throughput.yml is self-contained: it applies the experiment env by recreating the container, benchmarks, and in an always() step reverts to the stable baseline (OLLAMA_FLASH_ATTENTION=0). So a kernel that panics — e.g. tensor-core MMA on Turing under CUDA 11.4 — is a recorded result ("unusable"), and the testbed is still left clean for the next run. Never chase a green result by dodging the crash; the crash is the finding.

models is sm37-only for the large models; on sm75 size to the ~5 GiB it has.

Size a model's context/batch fit (fit-map sweep)

A different question from "is this FA path faster": at what context does a model spill to CPU, and does a smaller num_batch pull it back onto the GPU? On the K80 (FA off) the Q·Kᵀ score buffer is sized by num_batch, so batch — not just context — sets the VRAM wall. test-report-sweep.yml records this as a fit map (the basis for the qwen35moe clamp in #440):

gh workflow run test-report-sweep.yml --ref <branch> \
  -f suite=none \
  -f fit_map_models='qwen3.6:35b' \
  -f fit_map_num_batches='512 256 128' \
  -f fit_map_contexts='8192 65536 98304 131072 196608 262144'

suite=none skips the default throughput/MCP sweeps so only the fit-map runs (the fit-map is gated on fit_map_models, independent of suite). Per model it bounds the context ladder to the trained window and gates on tool support (cli.ts model-bounds, via /api/show), then loops num_batch x bounded-context. The report-sweep-results artifact's SUMMARY.md holds the fit-map table: per cell the fit (✅ on GPU / ⚠️ CPU spill / ❌ OOM), decode tok/s, total_s, per-die VRAM, active dies, and offload% — captured from the MCP GPU snapshot independent of the correctness verdict, so a judge hiccup never loses the fit data. Leave fit_map_models empty (default) and the sweep runs its normal throughput/MCP pass unchanged.