trt-cpp-runtime-quickstart

name	trt-cpp-runtime-quickstart
description	Load and run a TensorRT engine (.plan / .engine) from C++ using the TensorRT 11 / 10.x **modern Runtime API**, avoiding the deprecated TRT 8.x binding-index APIs that older guidance still promotes. Use whenever the user asks about loading or running a TensorRT .plan/.engine from C++, even on "minimal example" requests — without this skill the default reply uses deprecated enqueueV2-style code. Also use when the user hits "Engine plan file is generated on an incompatible device", deserializeCudaEngine returns nullptr, gets an enqueueV2 / IStreamReader deprecation warning, or wants to stream a .plan via IStreamReaderV2. Triggers: TensorRT C++ inference, load TensorRT plan C++, run .plan from C++, IRuntime example, deserializeCudaEngine, enqueueV3, enqueueV2 deprecated, setTensorAddress, getBindingIndex, IStreamReaderV2, libnvinfer C++. NOT for building engines (`trt-onnx-quickstart`), Python deploy, plugins, multi-GPU.
license	Apache-2.0
metadata	{"author":"NVIDIA Corporation","version":"1.0","tags":["tensorrt","cpp","inference","deployment","runtime"]}

TensorRT C++ Runtime Deploy

Load a serialized TensorRT engine from disk and run inference from C++ using only the modern Runtime API. Produces a minimal, copy-pasteable deploy harness that drops next to any .plan / .engine file and extends to production.

Reference samples to open before writing new code:

• quickstart/SemanticSegmentation/tutorial-runtime.cpp — cleanest minimal load-and-run example. Mirrors Steps 1–7 below.
• samples/sampleOnnxMNIST/sampleOnnxMNIST.cpp — end-to-end sample that also builds the engine; the runtime portion shows realistic I/O wiring.
• Public headers: include/NvInferRuntime.h — read IRuntime, ICudaEngine, IExecutionContext, IStreamReaderV2.

When to Use

Situation	Use this skill?
You have a .plan/.engine and need to run it from a C++ binary	Yes
You need a minimal harness that uses enqueueV3 + setTensorAddress	Yes
You want to load an engine from a std::istream or large file via IStreamReaderV2	Yes
You need to wire dynamic shapes (setInputShape) before inference	Yes
You are building / optimizing the engine (calibration, INT8, sparsity, builder configs)	No - use trtexec or IBuilder directly
You are deploying in Python	No - use tensorrt Python bindings
You are writing a plugin (IPluginV3) or custom layer	No - separate plugin skill
You need multi-GPU, MPS, MIG, or process-level orchestration	No - out of scope

Prerequisites

1. TensorRT installed. Verify NvInferRuntime.h is on the include path and libnvinfer.so is on the link path. On a TRT dev container these are in /usr/include/x86_64-linux-gnu/ and /usr/lib/x86_64-linux-gnu/ (or /opt/tensorrt/... for tarball installs).
2. CUDA toolkit available. cuda_runtime_api.h and libcudart.so must be reachable; nvcc --version should match the CUDA version the engine was built against.
3. A serialized engine. A .plan/.engine file built on the same major TRT version and the same GPU architecture (compute capability) you will deploy on. Engines are not portable across major TRT versions or across SMs unless the builder was given --hardwareCompatibilityLevel.
4. The engine's I/O tensor names. Inspect with:

   trtexec --loadEngine=model.plan --verbose 2>&1 | grep -E 'Input|Output'
   

5. A C++17 compiler (g++ >= 9 or clang++ >= 10).

Step 1: Create the IRuntime

The runtime owns engine deserialization and must outlive every ICudaEngine it creates. Construct one per process for typical deployments.

class Logger : public nvinfer1::ILogger {
public:
    void log(Severity severity, char const* msg) noexcept override {
        if (severity <= Severity::kWARNING) {
            std::cerr << msg << std::endl;
        }
    }
};

Logger gLogger;
std::unique_ptr<nvinfer1::IRuntime> runtime{
    nvinfer1::createInferRuntime(gLogger)};
if (!runtime) throw std::runtime_error("createInferRuntime failed");

A custom logger is mandatory - TensorRT does not log internally. Keep it process-global so deserialization warnings (version skew, calibrator mismatch) are not lost.

Step 2: Read the Plan into Memory

For small/medium engines (< ~1 GiB) read the whole file into a std::vector<char> and hand the pointer to IRuntime::deserializeCudaEngine(blob, size). This is what the SemanticSegmentation tutorial does and the simplest correct path:

std::ifstream f(planPath, std::ios::binary);
if (!f) throw std::runtime_error("cannot open " + planPath);
f.seekg(0, std::ios::end);
auto size = static_cast<size_t>(f.tellg());
f.seekg(0, std::ios::beg);
std::vector<char> blob(size);
if (!f.read(blob.data(), size))
    throw std::runtime_error("short read on " + planPath);

For very large engines, or when the bytes live behind a stream (HTTP, mmap'd archive, encrypted store), implement an IStreamReaderV2 - see Step 3.

Step 3 (optional): Use IStreamReaderV2 for Streaming Loads

IStreamReader (v1) is deprecated in TensorRT 11.0. Always use IStreamReaderV2: it reads into both host and device memory and is the only stream-reader form guaranteed for new code. Subclass and implement read(...) and seek(...):

class FileStreamReader : public nvinfer1::IStreamReaderV2 {
public:
    explicit FileStreamReader(std::string const& path)
        : mFile(path, std::ios::binary) {
        if (!mFile) throw std::runtime_error("open failed: " + path);
    }
    int64_t read(void* dst, int64_t n,
                 cudaStream_t /*stream*/) noexcept override {
        mFile.read(static_cast<char*>(dst), n);
        return mFile.gcount();
    }
    bool seek(int64_t off, nvinfer1::SeekPosition where) noexcept override {
        auto dir = (where == nvinfer1::SeekPosition::kSET) ? std::ios::beg
                 : (where == nvinfer1::SeekPosition::kCUR) ? std::ios::cur
                 : std::ios::end;
        mFile.clear();
        mFile.seekg(off, dir);
        return static_cast<bool>(mFile);
    }
private:
    std::ifstream mFile;
};

FileStreamReader rd{planPath};
std::unique_ptr<nvinfer1::ICudaEngine> engine{
    runtime->deserializeCudaEngine(rd)};

Step 4: Deserialize and Create an Execution Context

ICudaEngine is thread-safe for read-only queries; IExecutionContext is not - allocate one per inference thread.

std::unique_ptr<nvinfer1::ICudaEngine> engine{
    runtime->deserializeCudaEngine(blob.data(), blob.size())};
if (!engine) throw std::runtime_error("deserializeCudaEngine failed");

std::unique_ptr<nvinfer1::IExecutionContext> ctx{
    engine->createExecutionContext()};
if (!ctx) throw std::runtime_error("createExecutionContext failed");

Step 5: Wire Tensors with setTensorAddress

Enumerate I/O tensors via getNbIOTensors() + getIOTensorName(i). Use getTensorIOMode, getTensorDataType, and getTensorShape to size and allocate buffers. Set every tensor address before enqueueV3 - the modern API has no implicit binding-index map.

for (int i = 0; i < engine->getNbIOTensors(); ++i) {
    char const* name = engine->getIOTensorName(i);
    auto mode  = engine->getTensorIOMode(name);
    auto shape = engine->getTensorShape(name);   // -1 = dynamic dim
    if (mode == nvinfer1::TensorIOMode::kINPUT && hasDynamic(shape)) {
        // Fill in concrete shape, e.g. batch=1
        shape.d[0] = 1;
        ctx->setInputShape(name, shape);
    }
}
// After setInputShape on all dynamic inputs, query output shapes.
for (int i = 0; i < engine->getNbIOTensors(); ++i) {
    char const* name = engine->getIOTensorName(i);
    auto bytes = elementCount(ctx->getTensorShape(name))
               * dtypeSize(engine->getTensorDataType(name));
    void* dev = nullptr;
    cudaMalloc(&dev, bytes);
    ctx->setTensorAddress(name, dev);
}

Always call setInputShape for dynamic inputs before querying output shapes - the latter depends on the former.

Step 6: Run enqueueV3

enqueueV3(stream) is the only non-deprecated enqueue API; enqueueV2/execute* are gone in modern flows.

cudaStream_t stream{};
cudaStreamCreate(&stream);

cudaMemcpyAsync(devInput, hostInput, inBytes,
                cudaMemcpyHostToDevice, stream);
if (!ctx->enqueueV3(stream))
    throw std::runtime_error("enqueueV3 failed");
cudaMemcpyAsync(hostOutput, devOutput, outBytes,
                cudaMemcpyDeviceToHost, stream);
cudaStreamSynchronize(stream);

If you reuse buffers across iterations, skip the per-call setTensorAddress - addresses persist on the context until overwritten.

Step 7: Shutdown Order

Destroy in reverse construction order: contexts -> engines -> runtime, then free CUDA memory and destroy the stream. With std::unique_ptr this is automatic as long as the context is declared after the engine, and the engine after the runtime. Free cudaMalloc allocations explicitly (RAII wrapper recommended).

Build

Wire the steps above into your application's build system. For a standalone smoke test, a minimal build is:

g++ -std=c++17 runtime.cpp -o run -lnvinfer -lcudart   # adjust CUDA/TRT include + lib paths
./run model.plan

Common Errors

Symptom	Likely cause
deserializeCudaEngine returns nullptr, log says "version tag"	Engine built on a different TRT major version. Rebuild on the deploy version
nullptr with "engine plan file is generated on an incompatible device"	SM mismatch. Rebuild on the target SM or use --hardwareCompatibilityLevel
enqueueV3 returns false, log mentions "Tensor X has no address"	Forgot setTensorAddress for one of the I/O tensors
enqueueV3 false, "shape" in message	Forgot setInputShape for a dynamic input, or supplied an out-of-profile shape
cudaErrorIllegalAddress on H->D / D->H copy	Mismatched element count / dtype between host buffer and engine tensor
Process crashes inside TRT during destruction	Wrong destruction order - context outlived engine, or engine outlived runtime
cudaErrorMemoryAllocation during context creation	Workspace too big for the device; rebuild with smaller workspace

Pitfalls

• Do not use IStreamReader v1. Deprecated in TRT 11.0. Use IStreamReaderV2 (note cudaStream_t parameter on read).
• Do not use enqueueV2 / execute / binding indices. These are legacy paths; the only stable modern path is name-based setTensorAddress + enqueueV3.
• One IExecutionContext per thread. Sharing contexts across threads is undefined behavior; sharing the engine is fine.
• Stream lifetime. The CUDA stream passed to enqueueV3 must outlive the inference. Destroying it while work is in flight crashes or corrupts output.
• Async vs sync copies. Mixing synchronous cudaMemcpy with enqueueV3 on a stream serializes the GPU; always pair enqueueV3 with cudaMemcpyAsync on the same stream.
• Engine portability. A .plan is tied to (TRT major version, GPU SM, CUDA major version). Never check engines into a repo without recording these three facts.
• Logger lifetime. The logger passed to createInferRuntime must outlive the runtime; a stack-local logger in main is fine, a function- scope local is a use-after-free.
• Refit / weight streaming. Engines built with refit or weight streaming enabled need extra setup calls (setWeightStreamingBudgetV2, IRefitter); out of scope here.