tensorrt-llm

name	tensorrt-llm
description	Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.
version	1.0.0
author	Orchestra Research
license	MIT
dependencies	["tensorrt-llm","torch"]
platforms	["linux","macos"]
metadata	{"sonic":{"tags":["Inference Serving","TensorRT-LLM","NVIDIA","Inference Optimization","High Throughput","Low Latency","Production","FP8","INT4","In-Flight Batching","Multi-GPU"]}}

TensorRT-LLM

NVIDIA's open-source library for optimizing LLM inference with state-of-the-art performance on NVIDIA GPUs.

When to use TensorRT-LLM

Use TensorRT-LLM when:

• Deploying on NVIDIA GPUs (A100, H100, GB200)
• Need maximum throughput (24,000+ tokens/sec on Llama 3)
• Require low latency for real-time applications
• Working with quantized models (FP8, INT4, FP4)
• Scaling across multiple GPUs or nodes

Use vLLM instead when:

• Need simpler setup and Python-first API
• Want PagedAttention without TensorRT compilation
• Working with AMD GPUs or non-NVIDIA hardware

Use llama.cpp instead when:

• Deploying on CPU or Apple Silicon
• Need edge deployment without NVIDIA GPUs
• Want simpler GGUF quantization format

Quick start

Installation

# Docker (recommended)
docker pull nvidia/tensorrt_llm:latest

# pip install
pip install tensorrt_llm==1.2.0rc3

# Requires CUDA 13.0.0, TensorRT 10.13.2, Python 3.10-3.12

Basic inference

from tensorrt_llm import LLM, SamplingParams

# Initialize model
llm = LLM(model="meta-llama/Meta-Llama-3-8B")

# Configure sampling
sampling_params = SamplingParams(
    max_tokens=100,
    temperature=0.7,
    top_p=0.9
)

# Generate
prompts = ["Explain quantum computing"]
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(output.text)

Serving with trtllm-serve

# Start server (automatic model download and compilation)
trtllm-serve meta-llama/Meta-Llama-3-8B \
    --tp_size 4 \              # Tensor parallelism (4 GPUs)
    --max_batch_size 256 \
    --max_num_tokens 4096

# Client request
curl -X POST http://localhost:8000/v1/chat/completions \
  -H  \
  -d