Skip to main content

edge-llm-benchmark-performance

Benchmark TensorRT Edge-LLM serving performance (TTFT, inter-token latency, request latency, output throughput, concurrency scaling) through the OpenAI-compatible server with AIPerf. Use for "benchmark performance", "measure TTFT/ITL/tokens per second", prefill or decode sweeps over input and output lengths, in-flight-batching concurrency scaling, or comparing two deployments (quantization, speculative decoding, engine profile, another runtime). Do not use to launch a server only (edge-llm-serve), to score dataset accuracy (edge-llm-benchmark-accuracy), or to profile kernels.

Source facts

Repository
NVIDIA/TensorRT-Edge-LLM
Last source activity
November 13, 2026 at 06:10
Detected SKILL.md language
English
Stars
631
Forks
153

Install options

The review-first prompt is selected by default. You can switch to a direct command or download a local copy.

Review the source files

Read SKILL.md and any companion files shown by SkillsMP before deciding whether to install.

Showing SKILL.md

SKILL.md
Source instructions · Read-only preview
nameedge-llm-benchmark-performance
descriptionBenchmark TensorRT Edge-LLM serving performance (TTFT, inter-token latency, request latency, output throughput, concurrency scaling) through the OpenAI-compatible server with AIPerf. Use for "benchmark performance", "measure TTFT/ITL/tokens per second", prefill or decode sweeps over input and output lengths, in-flight-batching concurrency scaling, or comparing two deployments (quantization, speculative decoding, engine profile, another runtime). Do not use to launch a server only (edge-llm-serve), to score dataset accuracy (edge-llm-benchmark-accuracy), or to profile kernels.