| name | areno-profile-performance |
| description | Measure and diagnose AReno rollout, prefill, decode, training, checkpoint, role-switch, communication, or Python scheduling performance. Use when throughput or step time is slow and evidence from metrics, py-spy, or Nsight is required. Do not optimize before correctness is established. |
Profile AReno Performance
Profile a bounded, representative train or serve workload. Collect low-overhead
time, GPU, process, and TensorBoard evidence first; use sampling or tracing only
after those signals identify the subsystem to inspect.
python .agents/skills/areno-profile-performance/scripts/monitor_gpu.py \
--pid <areno-pid> --duration 60 --output /tmp/areno-gpu.jsonl
python .agents/skills/areno-profile-performance/scripts/monitor_process.py \
--pid <areno-pid> --duration 60 --output /tmp/areno-process.jsonl
python .agents/skills/areno-profile-performance/scripts/summarize_monitor.py \
/tmp/areno-gpu.jsonl
python .agents/skills/areno-profile-performance/scripts/summarize_monitor.py \
/tmp/areno-process.jsonl
python .agents/skills/areno-profile-performance/scripts/summarize_events.py \
<metrics-dir> --list
python .agents/skills/areno-profile-performance/scripts/summarize_events.py \
<metrics-dir> --pattern 'time/*' --pattern '*throughput*' --drop-first 1
python .agents/skills/areno-profile-performance/scripts/probe_openai_latency.py \
--base-url http://127.0.0.1:8000 --model <model> --requests 16 --concurrency 4
python .agents/skills/areno-profile-performance/scripts/build_nsys_command.py \
--output /tmp/areno-profile -- <bounded-command> [args...]
Workflow
- Record workload, commit, model, topology, token lengths, concurrency, GPU, and dependency versions.
- Find the parent AReno PID and monitor its process tree. For train jobs, attach during at least two post-warmup steps. For serve jobs, use a fixed request set and concurrency.
- Capture per-GPU utilization, memory, power, and target-process memory. Memory capacity, compute utilization, and throughput are separate signals.
- For train, list TensorBoard scalar names before selecting series. Summarize stage time, tokens/throughput, communication, optimizer, loss, and memory metrics that actually exist; do not assume fixed tags.
- For serve, measure TTFT and total request latency with streaming enabled while the GPU and process monitors run. Record prompt/output lengths and active concurrency.
- Exclude initialization, checkpoint load, compilation, and CUDA graph capture from steady-state conclusions, but report them separately when startup is the problem.
- Use
py-spy record -p <pid> -o /tmp/areno.svg --duration 30 for Python scheduling, data processing, serialization, blocking I/O, or compilation orchestration.
- Use a bounded Nsight Systems capture for GPU compute, communication, synchronization, allocator activity, or launch gaps. Read references/profile-policy.md.
- Compare baseline and candidate with identical workloads and multiple steady-state observations, then re-run correctness validation after optimization.
Report raw workload metadata, selected window, GPU peak/average memory and utilization, process CPU/RSS, stage breakdown or TTFT/latency, bottleneck evidence, profiler overhead, and before/after metrics. A single warmup step is not a benchmark.
