| name | run-aiperf-benchmark |
| description | Launches, monitors, debugs, and collects one run-scoped AIPerf Kubernetes benchmark without changing its workload semantics. Use after perf.yaml and aiperf-config.yaml have been configured for a smoke-tested Dynamo deployment. |
| license | Apache-2.0 |
| metadata | {"author":"NVIDIA","tags":["dynamo","aiperf","benchmarking"]} |
Run AIPerf Benchmark
Execute the configured Job and preserve operational evidence. Do not interpret performance.
Read agent-docs/rules/execution/deployment.md, agent-docs/rules/execution/logging.md,
agent-docs/rules/execution/run-artifacts.md, agent-docs/rules/benchmarking/benchmark-isolation.md,
agent-docs/rules/benchmarking/comparison-uncertainty.md,
agent-docs/rules/benchmarking/evidence-eligibility.md, agent-docs/rules/benchmarking/series-boundaries.md, and the
exact active benchmark plan first.
Preflight
- Confirm the selected Kubernetes context and namespace.
- Verify the plan path, SHA256, series ID, and performance question against the run-scoped configuration.
- Server-dry-run
perf.yaml and verify its Job name from metadata.name.
- Verify the frontend service is reachable in cluster and
/v1/models exposes the configured served model.
- Verify the AIPerf config, workload trace, tokenizer, PVC/mounts, image, and referenced secrets are available to the
benchmark pod.
- Confirm the AIPerf measurement is configured to finish in 30 minutes or less and uses one repetition by default.
- For an additional valid-run repetition, require the prior analysis to record why the repeat is necessary and worth
its GPU cost. Do not repeat only to obtain confidence intervals. The once-per-series noise-floor pilot (n=3)
required by
comparison-uncertainty.md is pre-authorized and arrives through the normal repeat_decision: necessary path with a rationale naming the series pilot; it needs no further justification.
- Confirm no other benchmark is targeting the same candidate.
- Every kubectl invocation this skill drives pins
--context "${KUBE_CONTEXT}" and -n "${NAMESPACE}",
bounds every wait with an explicit timeout, distinguishes job Failed from timeout (poll the job's true
condition rather than waiting on Complete alone), and chains commands so a failure never flows into the next
step — the same discipline deploy-dynamo-recipe's scripted blocks encode.
- Record neighbour occupancy in
benchmark_execution.json at run start and end (the pods scheduled on the serving
nodes, e.g. kubectl --context "${KUBE_CONTEXT}" get pods -A -o wide filtered to those nodes), so
comparison-uncertainty.md's like-for-like and transition checks have a recorded condition to read.
Execute And Monitor
- Apply the run-scoped Job.
- Wait with a bounded timeout and coarse polling.
- Monitor Job conditions, pod phase/restarts, recent events, and the relevant AIPerf log tail.
- On completion, copy the complete AIPerf output directory unchanged into
<DEPLOY_ROOT>/benchmark/raw_aiperf/ before deleting the Job or pod.
- Record exact commands, Job/pod names, start/end timestamps, configured and actual measurement duration, whether this
was the default run or an approved repeat, repeat rationale, AIPerf source/runtime versions, exit status, retry
history, and artifact paths in
benchmark_execution.json.
Debug By Ownership
| Failure | Action |
|---|
| Endpoint, model, or deployed workload failure | Record a handoff to recipe-deployer; do not patch the DGD here |
| Invalid trace or AIPerf settings | Return to configure-aiperf-benchmark |
| Repairable invalid evidence | Consume the audit blockers from analyze-aiperf-results, rerun the active series unchanged without overwriting prior raw artifacts, record the retry, then return to analyze-aiperf-results |
| Job scheduling, mount, image, or artifact-copy issue | Repair only the run-scoped benchmark manifest and record why |
| AIPerf client/tool failure | Use the pinned AIPerf docs/source, repair without changing benchmark semantics |
| Repeated identical failure | Stop and record the blocker |
Do not reduce request count, remove difficult trace rows, relax SLOs, lower load, switch schedule mode, or alter
ISL/OSL to make a failed benchmark finish. Such a change creates a different benchmark plan and comparison series.
Do not launch another valid run because a small delta was classified as noise or because confidence intervals are
absent.
Output Status
Set benchmark_execution.json.status to completed, failed, or blocked. A completed Job is not automatically a
valid benchmark; validity belongs to the audit phase of analyze-aiperf-results. Record the exact benchmark-plan path,
SHA256, series ID, and performance question in the execution record.
Keep logs only when they explain a failure beyond the concise execution ledger. Do not retain routine successful pod
logs or broad cluster snapshots.
