| name | run-guidellm-load-test |
| metadata | {"author":"rhoai3-coding-demo","version":"1.1.0","platform-family":"rhoai","platform-baseline":"repo","ocp-baseline":"repo","skill-group":"Demo Environment"} |
| description | Run an on-demand GuideLLM load test against a Stage 040 MaaS-published model endpoint. Use when the user asks to benchmark a private model, generate MaaS traffic, compare model latency or throughput, rerun a GuideLLM test with different prompt/rate/token settings, or collect short load evidence for Grafana and MaaS observability. Do NOT use for Stage 020/030 GPU recovery (use resume-gpu-demo), broad cluster troubleshooting (use rhoai-troubleshoot), or generic stage validation unless the user explicitly wants the GuideLLM load path exercised.
|
Run GuideLLM Load Test
Use the Stage 040 wrapper script as the source of truth. Do not recreate the Kubernetes Job manually unless the script itself is being debugged.
Preconditions
- Confirm
oc whoami succeeds against the intended OpenShift cluster.
- Confirm Stage 040 is deployed and the MaaS Gateway hostname is not a placeholder.
- Confirm the target model is ready through MaaS.
- Confirm a MaaS API key is available through
GUIDELLM_API_KEY, MAAS_API_KEY, or the kai-api-keys Secret created by Stage 080.
Never print, summarize, or store API key values. The wrapper creates a temporary in-cluster Secret, deletes it on exit, and stores only a safe console summary in a labeled ConfigMap.
Default Command
Run the default short test against the default model:
./stages/040-governed-models-as-a-service/run-guidellm-load-test.sh
The default is intentionally small for demo environments:
- Model:
nemotron-3-nano-30b-a3b
- Profile:
constant
- Rate:
1
- Maximum duration:
20 seconds
- Prompt samples:
5
- Output tokens:
64
Common Variants
Compare both local models:
./stages/040-governed-models-as-a-service/run-guidellm-load-test.sh qwen3-6-35b-a3b
./stages/040-governed-models-as-a-service/run-guidellm-load-test.sh nemotron-3-nano-30b-a3b
Run a small custom prompt:
GUIDELLM_REQUESTS=3 \
GUIDELLM_OUTPUT_TOKENS=32 \
GUIDELLM_PROMPT="Explain governed model access for enterprise AI teams." \
./stages/040-governed-models-as-a-service/run-guidellm-load-test.sh
Increase load carefully:
GUIDELLM_PROFILE=constant \
GUIDELLM_RATE=2 \
GUIDELLM_MAX_SECONDS=60 \
GUIDELLM_REQUESTS=30 \
GUIDELLM_OUTPUT_TOKENS=128 \
./stages/040-governed-models-as-a-service/run-guidellm-load-test.sh nemotron-3-nano-30b-a3b
Use a file or dataset supported by GuideLLM:
GUIDELLM_DATA=/path/to/prompts.jsonl \
./stages/040-governed-models-as-a-service/run-guidellm-load-test.sh qwen3-6-35b-a3b
Evidence To Capture
After the run, report:
- Target model and endpoint path, without secrets.
- Profile, rate, max duration, prompt sample count, and output token setting.
- Completed request count, incomplete request count, and error count from the GuideLLM console summary.
- Median or p95 request latency and token throughput if present in the console summary.
- Result
ConfigMap name.
Useful retrieval command:
oc get configmap -n maas -l app.kubernetes.io/name=guidellm-load-test
Inspect one result without exposing secrets:
oc get configmap <result-name> -n maas -o jsonpath='{.data.summary\.log}'
Safety Notes
- Keep the first run small. Increase
GUIDELLM_RATE, GUIDELLM_REQUESTS, or GUIDELLM_MAX_SECONDS only when the user asks for more load.
- Do not use raw GuideLLM JSON/CSV result files from inside the pod as shared artifacts; they can include backend arguments. Use the wrapper's stored console summary.
- If prerequisites are missing, the wrapper exits with code
2 and should be treated as a skipped load test, not a model failure.
- If the model is cold-starting after GPU resume, wait for Stage 030 validation before running larger tests.
