nvidia-tensorrt-llm-perf-host-analysis

Analyze host/CPU overhead in TensorRT-LLM inference from nsys traces. Detect whether host overhead is the bottleneck using GPU idle ratio, host prep exposed ratio, and per-phase evidence. For regressions, isolate forward steps via allreduce/NVTX patterns, compare host operation breakdowns across versions, and identify scheduling or request-management overhead. Supports optional inter-kernel gap, eager-vs-graph, pattern mapping, and multi-rank straggler drill-down. Use standalone or within perf-analysis. Triggers: host overhead, inter-step gap, scheduling overhead, forward step isolation, nsys iteration analysis, NVTX breakdown, request management overhead, GPU idle, host bottleneck, host prep exposed, inter-kernel gap, bubble analysis, graph coverage, eager kernel, rank imbalance, straggler detection.

Ir a la instalación

Repositorio: autohandai/community-skills
Última actividad en el origen: 30 de junio de 2026 a las 03:25
Idioma detectado de SKILL.md: inglés
Estrellas: 10
Forks: 3

Opciones de instalación

De forma predeterminada está seleccionado el prompt que primero revisa el origen. Puedes cambiar a un comando directo o descargar una copia local.

Instalar con Codex o Claude Copia este prompt, pégalo en Codex, Claude u otro asistente, y deja que revise la página de la skill y la instale por ti.

Un comando directo omite el prompt de revisión. Revisa el origen antes de ejecutarlo.

npx skills add https://github.com/autohandai/community-skills --skill nvidia-tensorrt-llm-perf-host-analysis

El comando permanece en una sola línea. Desplázate horizontalmente para revisarlo antes de copiarlo.

¿Prefieres una copia local? Descarga los archivos que SkillsMP tiene disponibles ahora.

Revisa los archivos de origen

Lee SKILL.md y los archivos complementarios que muestra SkillsMP antes de decidir si quieres instalarlo.

Explorador de archivos

13 archivos

SKILL.md

Instrucciones de origen · Vista previa de solo lectura

Volver a la búsqueda Volver a la instalación

name	nvidia-tensorrt-llm-perf-host-analysis
description	Analyze host/CPU overhead in TensorRT-LLM inference from nsys traces. Detect whether host overhead is the bottleneck using GPU idle ratio, host prep exposed ratio, and per-phase evidence. For regressions, isolate forward steps via allreduce/NVTX patterns, compare host operation breakdowns across versions, and identify scheduling or request-management overhead. Supports optional inter-kernel gap, eager-vs-graph, pattern mapping, and multi-rank straggler drill-down. Use standalone or within perf-analysis. Triggers: host overhead, inter-step gap, scheduling overhead, forward step isolation, nsys iteration analysis, NVTX breakdown, request management overhead, GPU idle, host bottleneck, host prep exposed, inter-kernel gap, bubble analysis, graph coverage, eager kernel, rank imbalance, straggler detection.
license	Apache-2.0
metadata	{"author":"NVIDIA Corporation","tags":["analysis","profiling","host-overhead","nsys","inference"],"dependencies":["workload-instrumentation","trace-interpretation"]}

Host Performance Analysis

Analyze host/CPU overhead in TensorRT-LLM inference workloads from nsys traces. This skill operates in two phases:

Phase	Question	Input	Output
Detection	Is host overhead the bottleneck?	Single nsys trace	YES/NO verdict with metric evidence
Root Cause

#	Metric	Threshold	What it answers
M1	GPU idle ratio	> 0.30	Is the GPU starved for work?
M2	Launch overhead ratio	> 0.10	Is kernel launch itself expensive?
M3a	Host prep exposed ratio	> 0.50	How well is host prep pipelined?
M3b	Host prep perf impact	> 0.05	How much throughput does exposed prep cost?
M3c	Host prep idle attribution	> 0.50	Is host prep the main cause of GPU idle?
M4	GPU utilization	< 0.60	Is GPU utilization too low?
M5	NCCL ratio (caveat)	> 0.20	Is communication a confounding factor?

Technique	Question	Key Output
Inter-Kernel Gap Analysis	Where is the GPU idle between kernels?	Gap bucket distribution, top-N largest gaps with source mapping
Eager vs Graph Classification	What fraction of kernels are graph-captured?	Graph coverage ratio, list of eager kernels with source attribution
Repeating-Pattern Mapping	Which functional group within a layer has the most overhead?	Per-group gap totals, priority ranking
Straggler Detection	Is one rank consistently slower?	Straggler rank ID, root cause (extra host work, queue depth feedback loop)

Finding	Optimization Pattern
Large gaps from Python tensor view chains	CUSTOM_OP — replace with C++ custom op
Graph-capturable kernels running eagerly	GRAPH_EXPAND — fix partition poisoning
Monolithic custom op blocking graph capture	GRAPH_SPLIT — split into capturable + eager parts
Host-device sync (`.item()`) in per-layer code	SYNC (Pattern 1: pre-compute on CPU) + HOIST (Variant B: pass from step level)
Per-layer buffer allocation	ALLOC — pre-allocate at init
Straggler rank with extra host work	Apply targeted optimization to coordinator-only code paths

File	Contents
references/metrics.md	Full metric definitions, formulas, SQL queries, M3 sub-metric analysis
references/thresholds.md	Aggregate and per-phase threshold tables
references/phase-classification.md	NVTX marker parsing, iteration classification, per-phase aggregation
references/output-format.md	Report template and integration JSON schema
references/examples.md	Worked scenarios (aggregate, phase-specific, and case study)
references/iteration-isolation-techniques.md	Allreduce, NVTX, and kernel-density iteration isolation techniques
references/trtllm-nvtx-ranges.md	TRT-LLM NVTX range reference with per-operation timings
references/kernel-level-analysis.md	Kernel-level drill-down techniques: gap analysis, graph classification, pattern mapping, straggler detection
references/nsys-schema.md	nsys SQLite schema reference and useful queries
scripts/analyze_host_overhead.py	Python script for Phase 2 root cause analysis
scripts/detect_host_overhead.py	Python script for Phase 1 detection verdict

nvidia-tensorrt-llm-perf-host-analysis

Datos de origen

Opciones de instalación

Revisa los archivos de origen

Más de este repositorio

Más de este repositorio

Host Performance Analysis

Host Performance Analysis

When to Use

Prerequisites

Key Concepts

Host Overhead in LLM Inference

Hidden vs Exposed Host Overhead

Forward Step Isolation

Phase Classification (Context vs Generation)

Phase 1: Detection (YES/NO Verdict)

Detection Metrics

Detection Workflow

Step 1: Input Validation

Step 2: Run Detection Script

Step 3: Interpret Verdict

Phase 2: Root Cause Analysis

Principles

Root Cause Workflow

Step 1: Collect nsys Traces

Step 2: Export to SQLite

Step 3: Run Host Overhead Analysis

Step 4: Interpret Results

Reading the Output

Step 5: Kernel-Level Drill-Down (Optional)

Kernel-Level Techniques

Workflow

Kernel-Level Findings to Optimization Patterns

Common Patterns and Root Causes

Pattern 1: Request Management Refactor

Pattern 2: Increased Kernel Launch Count

Pattern 3: New Bookkeeping Operations

Pattern 4: Flashinfer JIT Warmup Masquerading as Inference

Pattern 5: Context-Only Bottleneck (Masked by Aggregate)

Pitfalls

1. shortName is an Integer ID

2. NVTX textId vs text

3. Duplicate NVTX Ranges from TP Ranks

4. Negative Inter-Step Gaps

5. Benchmark Window Selection

Handoff to Optimization

Reference