deepstream-dev

name	deepstream-dev
version	1.3
last_updated	"2026-07-11T00:00:00.000Z"
tags	["nvidia","deepstream","gstreamer","tensorrt","video","vision"]
description	NVIDIA DeepStream SDK development guidance for Python pyservicemaker pipelines, video analytics, TensorRT integration, and streaming inference workflows.
license	CC-BY-4.0 AND Apache-2.0
compatibility	Guidance imported from the NVIDIA DeepStream 9.0 development skill for GStreamer and TensorRT video pipelines.

DeepStream Development Skill

When this skill is active, ALWAYS read the relevant reference documents before generating code. Do NOT rely on memory - the reference documents contain critical details about exact property names, correct API usage, and common pitfalls.

SDK and Architecture Quick Reference

DeepStream SDK 9.0 Version Requirements

• GStreamer: 1.24.2
• NVIDIA Driver: 590+
• CUDA: 13.1
• TensorRT: 10.14.1.48
• Platforms: Ubuntu 24.04 (x86_64 and ARM64/Jetson)

Typical Pipeline Flow

Source → Stream Muxer → Inference → [Tracker] → OSD → Renderer

Components in [brackets] are optional -- only add them when the user explicitly requests them.

Stage	Role	Key Element(s)	Required?
Source	Input from files, RTSP, cameras	nvurisrcbin (preferred), nvmultiurisrcbin, filesrc	Yes
Stream Muxer	Batches streams for inference	nvstreammux	Yes
Inference	TensorRT model execution	nvinfer, nvinferserver	Yes
Tracker	Multi-object tracking across frames	nvtracker	Only if requested
OSD	Draws bounding boxes, labels, overlays	nvosdbin	Yes (for visualization)
Renderer	Display or save output	nveglglessink, nv3dsink, filesink	Yes

Memory Model

DeepStream uses NVIDIA Video Memory Manager (NVMM) for zero-copy GPU buffer transfers. Caps strings use memory:NVMM to indicate GPU memory (e.g., video/x-raw(memory:NVMM), format=NV12).

Critical Rules

1. Only Add Requested Components: Do NOT add pipeline elements the user did not ask for.

   • Tracker (nvtracker): Only add when the user explicitly requests tracking or object IDs across frames
   • Secondary GIEs: Only add when the user requests classification or attribute extraction
   • Analytics (nvdsanalytics): Only add when the user requests line crossing, ROI counting, etc.
   • Message broker (nvmsgbroker/nvmsgconv): Only add when the user requests Kafka/cloud messaging
   • When in doubt, build the minimal working pipeline and let the user ask for additions

2. Default to nvurisrcbin for Sources: When the user says "camera", "stream", "video", or provides a file path:

   • Always use nvurisrcbin -- it handles RTSP, HTTP, and local files (file://) transparently
   • Only use filesrc + qtdemux + parser when the user explicitly needs raw file source control
   • For RTSP/live sources, also set live-source=1 on nvstreammux and sync=0 on the sink
   • Convert local paths to URI: "file://" + os.path.abspath(path)

3. Metadata Iteration: Use .frame_items and .object_items (returns iterators, NOT lists)

   • NEVER use len() on these - iterate to count
   • Iterator can only be consumed once

4. Request Pad Syntax: Use "sink_%u" template, NEVER literal pad names

   pipeline.link(("decoder", "mux"), ("", "sink_%u"))  # CORRECT
   # pipeline.link(("decoder", "mux"), ("", "sink_0"))  # WRONG - will fail
   

5. Platform Detection for Sinks:

   import platform
   sink_type = "nv3dsink" if platform.processor() == "aarch64" else "nveglglessink"
   

6. Buffer Cloning: Always clone buffers for async processing

   tensor = buffer.extract(0).clone()  # CRITICAL
   

7. Queue Types:

   • queue.Queue → Use with threading.Thread
   • multiprocessing.Queue → Use with multiprocessing.Process
   • Using wrong type causes silent data loss!

8. nvinfer Config Format:

   • YAML: Use property: section (NOT model:), key: value with space after colon
   • INI: Use [property] section, key=value with equals sign
   • Section MUST be named property

9. nvmsgbroker is a SINK: Cannot have downstream elements - use tee to split pipeline

10. ALL Sinks Need async=0 for Tee Splits or Dynamic Sources: CRITICAL for state transitions

    # When using tee splits OR dynamic sources, ALL sinks MUST have async=0
    pipeline.add("nveglglessink", "sink", {
        "sync": 0, "qos": 0,
        "async": 0  # CRITICAL - prevents state transition deadlock
    })
    

    Symptom if missing: Pipeline stays in PAUSED state, no video displays.

11. Built-in Probe Attachment: measure_fps_probe can only be attached to processing elements (e.g., nvinfer, nvosdbin), NOT to sink elements. Attaching to a sink raises RuntimeError: Probe failure.

12. Dynamic ONNX Models Require infer-dims: When the ONNX model has dynamic input shapes (e.g., exported with dynamic=True in Ultralytics YOLO, or with dynamic batch/height/width axes), you MUST add infer-dims=C;H;W to the nvinfer config. Without it, TensorRT sees -1 for dynamic dimensions and fails with setDimensions: Error Code 3. Common values:

    • YOLO models (640 input): infer-dims=3;640;640
    • Models with 416 input: infer-dims=3;416;416
    • Models with 1280 input: infer-dims=3;1280;1280

13. Ultralytics YOLO Output Format Depends on Model Generation — newer models (v10+/v26+) output post-NMS results; older models (v8/v11) output raw pre-NMS tensors. The custom parser and cluster-mode must match the actual output:

Model generation	Output tensor shape	Fields	cluster-mode
v8 / v11	[batch, 84, 8400]	[features(4+80), anchors] — raw cx/cy/w/h + class scores, no NMS	2 (NMS)
v10 / v26+	[batch, 300, 6]	[max_det, (x1,y1,x2,y2,conf,cls)] — already post-NMS, pixel coords	4 (none)

How to identify at runtime: log inferDims.d[0] and inferDims.d[1] inside the custom parser.

• d={84, 8400} → pre-NMS (v8/v11 style)
• d={300, 6} → post-NMS (v10/v26+ style)

Symptom of mismatch: If cluster-mode: 2 is used with a post-NMS [N, 6] output, bounding boxes appear shifted by 45° or 135° from the actual objects (DeepStream's NMS incorrectly re-processes already-final coordinates). If you see tilted or rotated boxes, also check the OBB / rotation_angle note in references/nvinfer_config.md: for non-OBB models, value-initialize NvDsInferObjectDetectionInfo with obj{} and keep rotation_angle = 0; plain NvDsInferObjectDetectionInfo obj; leaves fields uninitialized.

14. Virtual Environment Must Include pyservicemaker: pyservicemaker is installed system-wide but is NOT accessible from a standard Python virtual environment. When a task requires a venv (e.g., for model download/conversion pip dependencies), always install pyservicemaker and pyyaml inside the venv. The venv setup in generated code and README must always include:

    python3 -m venv venv
    source venv/bin/activate
    pip install /opt/nvidia/deepstream/deepstream/service-maker/python/pyservicemaker*.whl pyyaml
    pip install -r requirements.txt  # other dependencies
    

    Symptom if missing: ModuleNotFoundError: No module named 'pyservicemaker' when running the app inside the venv.

Key Paths (DeepStream 9.0)

• Models: /opt/nvidia/deepstream/deepstream/samples/models/
• Primary Detector: /opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx
• Tracker lib: /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
• Kafka lib: /opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so
• Sample configs: /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/

Reference Documents

IMPORTANT: Always read these documents for complete details. Do NOT generate code from memory.

Document	Use When
references/gstreamer_plugins.md	Looking up plugin properties, ALL properties listed
references/service_maker_api.md	Using Pipeline/Flow API, metadata access, probes, EventMessageUserMetadata
references/use_cases_pipelines.md	Building pipelines: simple playback, multi-inference, cascaded GIE
references/kafka_messaging.md	Kafka/message broker setup, nvmsgconv/nvmsgbroker config, msg2p-newapi
references/best_practices.md	Design patterns, common pitfalls, anti-patterns
references/buffer_apis.md	BufferProvider/Feeder (injection), BufferRetriever/Receiver (extraction)
references/media_extractor_advanced.md	MediaExtractor, MediaChunk, FrameSampler
references/utilities_config.md	PerfMonitor, EngineFileMonitor, SourceConfig, SensorInfo, SmartRecordConfig
references/nvinfer_config.md	nvinfer config file format, ALL parameters
references/tracker_config.md	nvtracker config, NvDCF/IOU/DeepSORT/NvSORT
references/troubleshooting.md	Error messages and solutions
references/rest_api_dynamic.md	REST API, dynamic source add/remove, nvmultiurisrcbin
references/metamux_config.md	nvdsmetamux config, parallel multi-model inference, metadata merging, source ID filtering
references/docker_containers.md	Docker images, Dockerfile examples, pyservicemaker install, container run commands

Quick Error Reference

Error	Solution
iterator has no len()	Iterate to count, don't use len()
pad template not found	Use "sink_%u" not "sink_0"
Queue data loss	Use multiprocessing.Queue with Process
Config parse failed	Use property: not model: in YAML
is-classifier deprecation warning	Use network-type: 1 instead of is-classifier: 1 for classifiers; omit both for detectors
min-boxes unknown key warning	Use minBoxes (camelCase) in class-attrs-* sections, not min-boxes
Secondary GIE inactive	Set process-mode: 2, check operate-on-gie-id
Tee/dynamic source stuck PAUSED	Set async: 0 on ALL sink elements
RTSP no data/reconnecting	Test URL with ffplay, check credentials
RuntimeError: Probe failure	measure_fps_probe cannot attach to sink elements; use nvinfer or nvosdbin instead
setDimensions negative dims / engine build failed	Add infer-dims=C;H;W for dynamic ONNX models (e.g., infer-dims=3;640;640)
No module named 'pyservicemaker' in venv	pip install /opt/nvidia/deepstream/deepstream/service-maker/python/pyservicemaker*.whl pyyaml inside the venv
AttributeError: object has no attribute 'obj_label'	Use obj_meta.label not obj_meta.obj_label in pyservicemaker (C API name differs from Python binding)

Anti-Patterns

• Adding optional pipeline elements the user did not request: It hides the minimal working path and complicates debugging.
• Using literal request-pad names, CPU-memory caps, or unsafe async buffer handling in GPU pipelines: These break DeepStream in non-obvious ways.
• Claiming a pipeline is correct without checking the actual platform, source type, and runtime metadata path.

Verification Protocol

Before claiming "skill applied successfully":

1. Pass/fail: The proposed pipeline only includes requested stages and uses the correct source, mux, inference, and sink conventions.
2. Pass/fail: Platform assumptions, NVMM usage, and request-pad or metadata handling are made explicit before code is declared ready.
3. Pass/fail: The answer includes a concrete runtime check path such as logs, rendered output, or message-broker validation.
4. Pressure-test scenario: Re-run the workflow on a Jetson target where the same pipeline shape needs different sink behavior and live-source settings.
5. Success metric: The user gets a minimal DeepStream pipeline that is reproducible and debuggable on the intended platform.

Cross-Client Portability

This skill is written to stay usable across GitHub Copilot, Claude Code, Codex, and Gemini CLI.

• GitHub Copilot: keep the folder in a Copilot-visible skill or plugin path, or wrap the workflow as project instructions if the host does not support portable skill folders directly.
• Claude Code: keep the folder in a local skills directory or a compatible plugin or marketplace source.
• Codex: install or sync the folder into $CODEX_HOME/skills/<skill-name> and restart Codex after major changes.
• Gemini CLI: this repository generates a project command named /skills:deepstream-dev from this skill. Rebuild commands with python scripts/export-gemini-skill.py deepstream-dev and then run /commands reload inside Gemini CLI.

MCP Availability And Fallback

Preferred MCP Server: None required

• Fallback prompt: "Use the deepstream-dev skill without MCP. Rely on the local SKILL.md, bundled references or scripts, and manual verification. Show the exact commands, evidence, and final checks you used before concluding."
• If the current host does not expose a matching server, use the bundled references, scripts, native toolchain, and manual workflow already described in this skill.
• Treat direct local verification, rendered output, logs, tests, or screenshots as the fallback evidence path before completion.

Related Skills

• development-workflow: Use it when the DeepStream task also needs a scoped implementation and verification plan.
• devops-tooling: Use it when the pipeline work also needs container, deployment, or CI handling.
• cloud-design-patterns: Use it when the video stack also needs broader streaming or distributed-system architecture tradeoffs.