nvidia-tensorrt-llm-perf-host-analysis

name	nvidia-tensorrt-llm-perf-host-analysis
description	Analyze host/CPU overhead in TensorRT-LLM inference from nsys traces. Detect whether host overhead is the bottleneck using GPU idle ratio, host prep exposed ratio, and per-phase evidence. For regressions, isolate forward steps via allreduce/NVTX patterns, compare host operation breakdowns across versions, and identify scheduling or request-management overhead. Supports optional inter-kernel gap, eager-vs-graph, pattern mapping, and multi-rank straggler drill-down. Use standalone or within perf-analysis. Triggers: host overhead, inter-step gap, scheduling overhead, forward step isolation, nsys iteration analysis, NVTX breakdown, request management overhead, GPU idle, host bottleneck, host prep exposed, inter-kernel gap, bubble analysis, graph coverage, eager kernel, rank imbalance, straggler detection.
license	Apache-2.0
metadata	{"author":"NVIDIA Corporation","tags":["analysis","profiling","host-overhead","nsys","inference"],"dependencies":["workload-instrumentation","trace-interpretation"]}

Host Performance Analysis

Analyze host/CPU overhead in TensorRT-LLM inference workloads from nsys traces. This skill operates in two phases:

Phase	Question	Input	Output
Detection	Is host overhead the bottleneck?	Single nsys trace	YES/NO verdict with metric evidence
Root Cause	What specifically regressed?	One or two nsys traces	NVTX per-step breakdown, regression sources, optional kernel-level drill-down

When to Use

• Before starting host optimization work -- confirms the bottleneck is real (Detection)
• As a sub-step of perf-analysis for bottleneck classification (Detection)
• When GPU utilization is suspiciously low and you need to know why (Detection)
• When throughput regressed but GPU kernel execution times are unchanged (Root Cause)
• When the gap between forward step iterations has increased (Root Cause)
• To compare inter-iteration overhead between two versions of the inference engine (Root Cause)
• When you need sub-operation granularity on inter-kernel gaps or graph coverage (Root Cause, kernel-level drill-down)
• When piecewise CUDA graph coverage is unexpectedly low (Root Cause, kernel-level drill-down)
• When multi-rank inference shows unexplained performance asymmetry (Root Cause, kernel-level drill-down)

Do NOT use when:

• The regression is in individual kernel performance (use perf-nsight-compute-analysis)
• You need to profile a workload from scratch (use workload-instrumentation first)
• The issue is NCCL communication (use distributed analysis)

Prerequisites

• An nsys trace file (.sqlite or .nsys-rep) from a TRT-LLM benchmark run
• For Root Cause comparison: two traces (baseline and target)
• Python 3 with sqlite3 support

---

Key Concepts

Host Overhead in LLM Inference

In an LLM inference loop, each iteration consists of:

[inter-step gap] -> [_forward_step] -> [inter-step gap] -> [_forward_step] -> ...

The forward step includes GPU kernel execution (GEMM, attention, normalization, allreduce) plus host-side preparation. The inter-step gap includes host-side work between forward steps (scheduling, request fetching, broadcasting, sampling, response handling).

See references/trtllm-nvtx-ranges.md for the full per-operation breakdown and timing ranges.

Hidden vs Exposed Host Overhead

Host overhead only hurts performance when it is exposed -- the GPU is idle waiting for work. When host prep overlaps with GPU execution, it is hidden and free. See references/metrics.md (M3 section) for diagrams and the exposed/hidden computation.

Forward Step Isolation

In TP configurations, forward steps are isolated via allreduce kernel grouping (deterministic count per transformer layer). For TP=1, NVTX _forward_step ranges are used directly. See references/iteration-isolation-techniques.md for the full algorithm.

Phase Classification (Context vs Generation)

Iterations are classified by NVTX marker text into context (eager, no CUDA graphs) and generation (CUDA graph replay). Per-phase analysis is critical because aggregate metrics can mask phase-specific bottlenecks. See references/phase-classification.md.

---

Phase 1: Detection (YES/NO Verdict)

Determine whether host overhead is the primary bottleneck.

Detection Metrics

Six metrics in four categories. See references/metrics.md for full definitions, formulas, and SQL queries.

#	Metric	Threshold	What it answers
M1	GPU idle ratio	> 0.30	Is the GPU starved for work?
M2	Launch overhead ratio	> 0.10	Is kernel launch itself expensive?
M3a	Host prep exposed ratio	> 0.50	How well is host prep pipelined?
M3b	Host prep perf impact	> 0.05	How much throughput does exposed prep cost?
M3c	Host prep idle attribution	> 0.50	Is host prep the main cause of GPU idle?
M4	GPU utilization	< 0.60	Is GPU utilization too low?
M5	NCCL ratio (caveat)	> 0.20	Is communication a confounding factor?

Host prep confirmation rule: Host prep is a confirmed bottleneck only when both M3b AND M3c cross their thresholds.

Thresholds are configurable with per-phase variants. See references/thresholds.md.

Detection Workflow

Step 1: Input Validation

# Accept .sqlite or .nsys-rep
ls -la <trace_file>

# If .nsys-rep, export to SQLite first
nsys export -t sqlite -o <output.sqlite> <input.nsys-rep>

Step 2: Run Detection Script

python scripts/detect_host_overhead.py \
  --trace /path/to/trace.sqlite \
  --output /path/to/verdict.json

The script computes M1, M2, M4, M5 from SQL, optionally M3 via range intersection, applies the verdict logic, and outputs structured JSON. See references/output-format.md for the output schema.

For manual metric extraction via SQL, see references/nsys-schema.md.

Step 3: Interpret Verdict

Overall Verdict:

if aggregate_verdict == YES or context_verdict == YES or generation_verdict == YES:
    overall_verdict = YES

Per-phase analysis can elevate the verdict but never demote it.

Format using the template in references/output-format.md.

Next Steps:

• If YES -> Proceed to Phase 2 (Root Cause) below, then use perf-host-optimization skill
• If NO -> Use perf-nsight-compute-analysis for kernel SOL% or trace-interpretation for full classification

---

Phase 2: Root Cause Analysis

Identify which specific host operations regressed and by how much. Works with a single trace (breakdown) or two traces (comparison).

Principles

1. Isolate forward steps, not the full trace. nsys traces contain warmup, JIT, model loading, and teardown.
2. Use structural kernel patterns for iteration detection. Allreduce grouping is more robust than kernel density.
3. Compare steady-state iterations. Filter to identical workload (same batch size, same ctx/gen mix).
4. Per-step metrics, not totals. Always compare per-step averages.

Root Cause Workflow

Step 1: Collect nsys Traces

Profile both versions (if comparing) with identical settings:

nsys profile -o /path/to/trace \
  -t cuda,nvtx,osrt \
  --force-overwrite=true \
  --cuda-memory-usage=true \
  -w true \
  <benchmark_command> --num_requests 500

Step 2: Export to SQLite

nsys export --type=sqlite --force-overwrite=true -o trace.sqlite trace.nsys-rep

Step 3: Run Host Overhead Analysis

# Two-trace comparison
python scripts/analyze_host_overhead.py \
  --baseline /path/to/baseline/trace.sqlite \
  --target /path/to/target/trace.sqlite \
  --baseline-label "v1.1" \
  --target-label "main" \
  --output /path/to/output/analysis.txt

# Single-trace breakdown
python scripts/analyze_host_overhead.py \
  --baseline /path/to/trace.sqlite \
  --baseline-label "current"

Step 4: Interpret Results

The script produces:

1. Allreduce-based iteration detection -- confirms forward step boundaries
2. Per-step wall time comparison -- quantifies the regression
3. NVTX per-step breakdown -- identifies which host operations regressed
4. GPU kernel comparison -- confirms GPU execution is unchanged
5. CUDA API comparison -- detects kernel launch overhead changes

Reading the Output

Per-Step Wall Time:

Avg wall time per step: 3,317 us (baseline) vs 3,978 us (target)  +19.9%

This is the primary regression metric.

NVTX Breakdown:

Operation           | baseline (us/step) | target (us/step) | Delta    | Status
_fetch_new_requests |               36   |             270  | +234     | REGRESSION
broadcast_requests  |                -   |             250  | +250     | NEW
_update_requests    |              413   |             723  | +310     | REGRESSION

Focus on operations with large absolute deltas.

GPU Kernel Comparison:

Kernels per step (launched): 6.2 (baseline) vs 21.9 (target)  +253%

More individual launches = more host-side launch overhead.

Step 5: Kernel-Level Drill-Down (Optional)

When the NVTX breakdown identifies a regressing operation but does not reveal why (the overhead is inside the GPU dispatch, not between NVTX ranges), drill below NVTX operations into individual GPU kernel launches.

See references/kernel-level-analysis.md for full technique details, SQL queries, and examples.

When to drill down:

• An operation has high wall time but the overhead is inside GPU dispatch, not between NVTX ranges
• You need to understand how much of the forward pass is graph-captured vs eager
• Per-layer overhead is significant and you need to map kernels to functional groups
• Multi-rank inference shows unexplained performance asymmetry

Kernel-Level Techniques

Technique	Question	Key Output
Inter-Kernel Gap Analysis	Where is the GPU idle between kernels?	Gap bucket distribution, top-N largest gaps with source mapping
Eager vs Graph Classification	What fraction of kernels are graph-captured?	Graph coverage ratio, list of eager kernels with source attribution
Repeating-Pattern Mapping	Which functional group within a layer has the most overhead?	Per-group gap totals, priority ranking
Straggler Detection	Is one rank consistently slower?	Straggler rank ID, root cause (extra host work, queue depth feedback loop)

Workflow

1. Start with Inter-Kernel Gap Analysis — bucket the gap distribution to understand the dominant overhead type (graph dispatch, Python interpreter, host-device sync)
2. If piecewise graph is in use, run Eager vs Graph Classification to measure graph coverage and identify unnecessary eager kernels
3. For per-layer overhead, use Repeating-Pattern Mapping to isolate the highest-overhead functional group within a single layer
4. For multi-rank setups, run Straggler Detection if per-step wall time varies across ranks

Kernel-Level Findings to Optimization Patterns

Finding	Optimization Pattern
Large gaps from Python tensor view chains	CUSTOM_OP — replace with C++ custom op
Graph-capturable kernels running eagerly	GRAPH_EXPAND — fix partition poisoning
Monolithic custom op blocking graph capture	GRAPH_SPLIT — split into capturable + eager parts
Host-device sync (.item()) in per-layer code	SYNC (Pattern 1: pre-compute on CPU) + HOIST (Variant B: pass from step level)
Per-layer buffer allocation	ALLOC — pre-allocate at init
Straggler rank with extra host work	Apply targeted optimization to coordinator-only code paths

---

Common Patterns and Root Causes

Pattern 1: Request Management Refactor

Symptom: _fetch_new_requests regressed 5-10x, new broadcast_requests operation. Cause: Request fetching refactored for multi-rank broadcasting in TP. Mitigation: Optimize broadcast path; batch request state updates.

Pattern 2: Increased Kernel Launch Count

Symptom: 3-5x more cudaLaunchKernel calls per step, similar GPU time. Cause: Operations that were fused or graph-captured are now individual launches. Mitigation: Re-fuse kernels; extend CUDA graph capture scope.

Pattern 3: New Bookkeeping Operations

Symptom: New NVTX ranges like _write_finish_reasons, handle_additional_outputs. Cause: New features added to the inference loop without overhead budgeting. Mitigation: Defer non-critical bookkeeping to async paths; batch updates.

Pattern 4: Flashinfer JIT Warmup Masquerading as Inference

Symptom: Massive elementwise/reduce kernel counts in "steady state" analysis. Cause: Analysis window includes flashinfer JIT compilation phase. Fix: Use allreduce-based iteration isolation, not kernel density or time windows.

Pattern 5: Context-Only Bottleneck (Masked by Aggregate)

Symptom: Aggregate metrics below threshold, but context iterations have 50% GPU idle. Cause: Generation iterations dilute the context-phase bottleneck. Fix: Per-phase analysis in Detection phase catches this.

---

Pitfalls

1. shortName is an Integer ID

In CUPTI_ACTIVITY_KIND_KERNEL, shortName is an integer referencing StringIds.id. Always join. See references/nsys-schema.md.

2. NVTX textId vs text

Most NVTX events have textId (integer) but NULL text. Join with StringIds. See references/nsys-schema.md.

3. Duplicate NVTX Ranges from TP Ranks

In TP configurations, each rank reports NVTX ranges independently. De-duplicate by grouping entries within 100us of each other.

4. Negative Inter-Step Gaps

When TP ranks report overlapping NVTX ranges, gap = next_start - prev_end can be negative. Use the maximum end time when de-duplicating.

5. Benchmark Window Selection

The allreduce-based window captures context+generation phases; steady-state NVTX filtering captures generation-only. Both are valid; use the appropriate one for your comparison goal.

---

Handoff to Optimization

When analysis is complete and the verdict is YES, hand off to the perf-host-optimization skill with:

1. Detection verdict and evidence: Which metrics crossed thresholds (M1-M5), whether host prep was confirmed (M3b+M3c), and per-phase breakdown.

2. NVTX-based triage (from Root Cause): Top regressing operations by absolute delta (us/step). Map NVTX range names to source functions -- see references/trtllm-nvtx-ranges.md.

3. Handoff data block: Include structured data from references/output-format.md (see "Handoff to Optimization" section).

4. Kernel-level findings (from drill-down, if performed): Inter-kernel gap distribution, graph coverage ratio, per-group overhead map, and straggler rank identification. Map findings to optimization patterns using the table in the Root Cause kernel-level drill-down section above.

---

Reference

File	Contents
references/metrics.md	Full metric definitions, formulas, SQL queries, M3 sub-metric analysis
references/thresholds.md	Aggregate and per-phase threshold tables
references/phase-classification.md	NVTX marker parsing, iteration classification, per-phase aggregation
references/output-format.md	Report template and integration JSON schema
references/examples.md	Worked scenarios (aggregate, phase-specific, and case study)
references/iteration-isolation-techniques.md	Allreduce, NVTX, and kernel-density iteration isolation techniques
references/trtllm-nvtx-ranges.md	TRT-LLM NVTX range reference with per-operation timings
references/kernel-level-analysis.md	Kernel-level drill-down techniques: gap analysis, graph classification, pattern mapping, straggler detection
references/nsys-schema.md	nsys SQLite schema reference and useful queries
scripts/analyze_host_overhead.py	Python script for Phase 2 root cause analysis
scripts/detect_host_overhead.py	Python script for Phase 1 detection verdict