rtvi-cv-scaffold-vss-service

# Construct RTVI VSS CV Service Scaffolds a deployable custom perception microservice that: 1. runs a DeepStream pipeline with **YOLO26** primary inference + tracker, 2. converts detection metadata to the VSS protobuf schema and publishes it to the **`mdx-raw`** Kafka topic, and 3. drops into the existing VSS compose stack via the `bp_developer_search_2d` and `bp_developer_alerts_2d_cv` profile flags so the downstream Search Workflow, Alert Verification, and Behavior Analytics services consume it without further changes. The scaffolded output is a runnable repo, not a design document. ## Scope boundaries This skill name describes the **service type** (a VSS-bound RTVI CV microservice), not a model family. What is generic vs YOLO26-specific: | Layer | Scope | |-------|-------| | VSS integration | Generic — compose profiles, host networking, `mdx-raw`, protobuf-2 payload, tests, smoketest | | Scaffold script output | **YOLO26 reference** — `pgie-yolo26-config.txt`, `YOLO26_*` mount paths, `NvDsInferParseYolo26` | | Other ONNX detectors | Adapt the scaffold manually, or use `rtvi-cv-customize-model` to swap the model inside the stock `vss-rt-cv` perception container | | Segmentation frame masks | Documented in `integration-contract.md`; not auto-generated by the scaffold — extend msgconv/wrapper after scaffolding | Do not treat the scaffold as a model-agnostic generator. Agents should either run the YOLO26 scaffold as-is or consciously edit pgie/compose paths for another detector while keeping the VSS contract fixed. ## When to use Use this skill when the user wants to: - replace VSS's default perception service (RT-DETR / GDINO / YOLOv11) with a custom YOLO26-based detector, - add new object classes or domain-specific tracking logic while keeping the VSS Search / Alerts / Behavior Analytics workflows intact, - ship a perception microservice on a customer's hardware that integrates with a VSS deployment they manage. Do **not** use this skill to: - deploy or operate VSS itself (use VSS deployment runbooks), - swap the detector in the default `vss-rt-cv` container without a new microservice (use `rtvi-cv-customize-model`). This skill assumes a target VSS deployment already exists or will be brought up separately. ## Instructions - Read [references/integration-contract.md](references/integration-contract.md) first and keep `mdx-raw`, `msg-conv-payload-type=2`, broker reachability, and the protobuf-2 contract fixed unless the user explicitly wants to break VSS compatibility. - Treat all VSS deployment paths as relative to a separate checkout of the public [VSS Blueprint repository](https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization), not the DeepStream repository. Clone or reuse a VSS checkout that is **v3.2.1 or compatible**, then deploy that release (see [VSS Quickstart](https://docs.nvidia.com/vss/latest/quickstart.html#download-the-deployment-package)). - If the user does **not** have the YOLO26 ONNX, labels file, or custom parser `.so`, say so plainly: scaffolding and unit tests can proceed with placeholders, but the live DeepStream app cannot run yet. Point them at Ultralytics/Hugging Face for pretrained weights and ONNX export (see [references/yolo26-deepstream.md](references/yolo26-deepstream.md)). - When assets are missing, stop the "live validation" path at scaffold generation plus host-only unit tests. Do not imply that `docker compose up`, `deepstream-app`, or `kafka_smoketest.py` can succeed without the customer-supplied ONNX, labels matching `num-detected-classes`, and a parser exposing `NvDsInferParseYolo26`. - Start the generated service only after the VSS Kafka topic initializer has completed. Keep it as a separate Compose application using host networking. Set `KAFKA_BOOTSTRAP` to Kafka's host-reachable advertised listener (default: `localhost:9092`). Compose cannot resolve `depends_on` across separate invocations. ## Examples - "Create a custom RTVI CV microservice that runs YOLO26 in DeepStream and publishes object metadata so VSS Search and Alerts can consume it." - "Validate the generated RTVI VSS service scaffold locally, then explain how to verify the live integration on a GPU host with a deployed VSS stack." - "I do not have a trained YOLO26 ONNX, labels file, or parser `.so` yet. Can the live VSS integration still run?" ## VSS source location This skill ships the scaffold generator, but not the VSS deployment. Clone the public VSS Blueprint repository separately or reuse an existing checkout, then set `VSS_ROOT` to that checkout explicitly. Do not search for or clone VSS relative to the generated service directory: ```bash # Customer-specific path to an existing VSS v3.2.1-compatible checkout. export VSS_ROOT=/absolute/path/to/video-search-and-summarization VSS_DEPLOY_DIR="${VSS_ROOT}/deploy/docker" test -f "${VSS_DEPLOY_DIR}/compose.yml" || { echo "Missing VSS compose file: ${VSS_DEPLOY_DIR}/compose.yml" >&2 exit 1 } echo "Using VSS deployment: ${VSS_DEPLOY_DIR}" ``` Relevant customer-accessible VSS locations include: - [`deploy/docker/scripts/dev-profile.sh`](https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization/blob/v3.2.1/deploy/docker/scripts/dev-profile.sh) — brings up Search or Alerts using the documented Quickstart flow - [`deploy/docker/developer-profiles/dev-profile-search/`](https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization/tree/v3.2.1/deploy/docker/developer-profiles/dev-profile-search) - [`deploy/docker/developer-profiles/dev-profile-alerts/`](https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization/tree/v3.2.1/deploy/docker/developer-profiles/dev-profile-alerts) - [`deploy/docker/services/rtvi/rtvi-cv/`](https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization/tree/v3.2.1/deploy/docker/services/rtvi/rtvi-cv) — stock RTVI-CV service definition The custom service generated by this skill remains in `<target-dir>` and runs as a separate, host-networked Compose application alongside VSS. Set `KAFKA_BOOTSTRAP` to Kafka's host-reachable advertised listener (default: `localhost:9092`). Do not look for generated YOLO26 files inside either the DeepStream or stock VSS checkout. ## Required reads Read these before generating code so the implementation matches the documented VSS data path rather than a generic Kafka producer: 1. [references/integration-contract.md](references/integration-contract.md) — the exact mdx-raw protobuf contract. For segmentation/frame-mask services, read its segmentation frame-mask payload section before implementing msgconv, wrapper, Kafka, or protobuf changes. 2. [references/yolo26-deepstream.md](references/yolo26-deepstream.md) — DeepStream nvinfer config for YOLO26. 3. [references/vss-profile-integration.md](references/vss-profile-integration.md) — how the service plugs into Search / Alerts / Behavior Analytics. 4. VSS Object Detection and Tracking — `https://docs.nvidia.com/vss/latest/object-detection-tracking.html` 5. VSS Behavior Analytics — `https://docs.nvidia.com/vss/latest/behavior-analytics.html` 6. VSS Search Workflow — `https://docs.nvidia.com/vss/latest/agent-workflow-search.html` 7. DeepStream `Gst-nvmsgconv` — `https://docs.nvidia.com/metropolis/deepstream/9.1/text/DS_plugin_gst-nvmsgconv.html` 8. DeepStream `Gst-nvmsgbroker` — `https://docs.nvidia.com/metropolis/deepstream/9.1/text/DS_plugin_gst-nvmsgbroker.html` ## Inputs to confirm Ask the user for these. Use placeholders if not provided — do not block. - `service-name` (slug; becomes container name and python package) - `yolo26-onnx-path` (host path to the YOLO26 ONNX export — not `.pt`) - `yolo26-labels-path` (host path to a one-class-per-line labels file) - `yolo26-parser-lib` (host path to the YOLO26 custom-parser `.so`, or the parser function name if compiled into the customer image) - `num-classes` (integer; must match the labels file) - `target-vss-profiles` — any subset of: - `bp_developer_search_2d` (Search Profile) - `bp_developer_alerts_2d_cv` (Alerts Profile + Behavior Analytics consumer) - `kafka-bootstrap` (Kafka's host-reachable advertised listener; default `localhost:9092`) - `kafka-topic` (default `mdx-raw` — VSS consumers expect this name) - `input-rtsp-uri` or input video file path ## Workflow 1. Read [references/integration-contract.md](references/integration-contract.md). The fixed parts of the contract (topic name, payload type, schema library) are not negotiable if the goal is to plug into an existing VSS deployment. For segmentation/frame-mask services, follow the segmentation payload contract in that reference before changing msgconv or wrapper code. 2. Scaffold the **YOLO26 reference** service (do not generalize pgie paths unless the user explicitly needs another detector): ```bash python3 scripts/scaffold_rtvi_vss_service.py \ --service-name <service-name> \ --output-dir <target-dir> \ --num-classes <N> \ --vss-profiles bp_developer_search_2d bp_developer_alerts_2d_cv ``` 3. Drop the customer's YOLO26 ONNX, labels file, and custom-parser library into the paths the generated `service_config.json` references. If those artifacts do not exist yet, stop after scaffolding and host-only unit tests; the live DeepStream service cannot start correctly without them. 4. Build the image: ```bash docker build -t <service-name>:dev <target-dir> ``` 5. From the separate VSS checkout, bring up the matching Search or Alerts profile using the VSS Quickstart and `deploy/docker/scripts/dev-profile.sh`. Confirm that VSS and its one-shot Kafka topic initializer are ready, then start the generated service as a separate Compose application: For Search or Alerts, find the matching one-shot initializer without assuming a Compose project or container name: ```bash docker ps -a \ --filter label=com.docker.compose.service=kafka-topic-init-container \ --format 'table {{.Names}}\t{{.Status}}' ``` Identify the selected deployment's initializer and confirm it is `Exited (0)`, then start the generated service with the matching gate: ```bash cd <target-dir> docker compose -f compose/service.compose.yml \ --profile bp_developer_search_2d up -d # Or for Alerts: docker compose -f compose/service.compose.yml \ --profile bp_developer_alerts_2d_cv up -d ``` Do not add a cross-file `depends_on` entry. The generated service uses `network_mode: host`. Set `KAFKA_BOOTSTRAP` to Kafka's host-reachable advertised listener (default: `localhost:9092`). The readiness check enforces startup order. 6. Verify the metadata flow: ```bash cd <target-dir> python3 tools/kafka_smoketest.py --describe-only python3 tools/kafka_smoketest.py --timeout 120 ``` The smoke test auto-detects Kafka only when exactly one matching container is running. If multiple deployments are active, pass the selected running name with `--kafka-container`. Its `--bootstrap-server` is resolved inside that Kafka container and normally remains `localhost:9092`, even when the detector's host-facing `KAFKA_BOOTSTRAP` uses a different port. A successful consume confirms at least one non-empty message arrived on `mdx-raw` within the timeout. It does not decode the protobuf payload or validate sensorId, objects, or bbox fields. 7. Confirm downstream pickup: - **Search Profile**: query the Video Analytics API for ingested events, run an embed query and an attribute query against the search workflow. - **Alerts Profile**: watch `mdx-incidents` for behavior-analytics output; the `alert-bridge` service should generate VLM-verified incidents. - **Behavior Analytics**: the selected VSS Search or Alerts deployment starts its own behavior consumer. This is part of the VSS stack, not another profile for the generated service. It consumes `mdx-raw` and emits behavior windows to `mdx-incidents`. ## Service shape (generated) The scaffolder emits this layout: ``` <service-name>/ ├── Dockerfile DeepStream 9.1 + custom parser hook ├── README.md customer-facing build/run/plug-in guide ├── service_config.json declarative config (topic, profile flags, paths) ├── compose/ │ └── service.compose.yml service def with profiles for VSS plug-in ├── pipeline/ │ ├── ds-app-config.txt deepstream-test5 derived; msgconv→msgbroker→mdx-raw │ ├── ds-start.sh container entrypoint │ └── configs/ │ ├── pgie-yolo26-config.txt nvinfer config skeleton for YOLO26 │ ├── tracker-nvdcf.yml NvDCF tracker config │ ├── cfg_kafka.txt librdkafka producer overrides │ ├── msgconv_config.txt mega2d sensor context │ └── labels.txt class labels placeholder ├── app/ │ ├── __init__.py │ ├── contracts.py NvDsEventMsgMeta-aligned event envelope │ ├── pipeline_plan.py declarative stage list (used by tests) │ └── service.py adapter helpers for extension/derived events ├── tools/ │ └── kafka_smoketest.py consumes one mdx-raw message and asserts it is non-empty └── tests/ ├── test_pipeline_config.py asserts the Kafka adapter contract and msgconv→msgbroker→mdx-raw wiring ├── test_service.py exercises the python adapter without GPU └── test_compose.py asserts profile gates, host networking, and liveness healthcheck ``` ## Non-negotiable contract elements The customer can change almost everything *except* these — they're what the VSS stack consumes: - Kafka topic `mdx-raw`, using the host-reachable advertised listener selected by `KAFKA_BOOTSTRAP` (default: `localhost:9092`). - `msg-conv-payload-type=2` (`NVDS_PAYLOAD_DEEPSTREAM_PROTOBUF`) with `msg-conv-msg2p-new-api=1` so the generated `deepstream-app` serializes frame/object metadata directly: protobuf serialized by `msg-conv-msg2p-lib` (`libnvds_msgconv_mega2d.so`, or `libnvds_msgconv.so` on DGX-SPARK/THOR); `libnvds_kafka_proto.so` is the `msg-broker-proto-lib` Kafka transport adapter only. - A protobuf `Frame` payload with `sensorId`, `timestamp`, and `objects[]` carrying `id`, `bbox`, `type`, `confidence`. This is what `vss-search-analytics-*`, `vss-behavior-analytics-*`, and `vss-video-analytics-api-*` deserialize. - `network_mode: host`. - An explicit deployment precondition that the VSS `kafka-topic-init-container` completed successfully before this separate Compose application starts. If any of these change, the customer is no longer plugging into VSS — they are running an isolated CV service. ## Validation Run unit tests on any host (no GPU required): ```bash cd <target-dir> python3 -m unittest discover -v -s tests -p 'test_*.py' ``` Run end-to-end on a host with GPU + a deployed VSS stack: ```bash cd <target-dir> python3 tools/kafka_smoketest.py --describe-only python3 tools/kafka_smoketest.py --timeout 120 ``` A successful smoke test confirms non-empty bytes arrived on `mdx-raw` within the timeout. It does not decode the protobuf payload or prove the message came from this service specifically. To verify downstream pickup, check the Search / Alerts / Behavior Analytics services via their own APIs. Do not present local unit-test success as proof that live VSS integration is ready. Without the customer ONNX, labels file, and parser library, the live DeepStream path remains blocked even if scaffolding and unit tests succeed. ## Implementation guardrails - Do not invent topic names. `mdx-raw` is the single perception ingress. Per-service prefixes break the existing VSS consumers. - Do not switch to `msg-conv-payload-type=0` or `=1`. VSS deserializers expect type 2. - Do not bridge through a custom Python Kafka producer. The DeepStream `nvmsgconv → nvmsgbroker` boundary is what produces correctly-framed protobuf with timestamps in nanoseconds. A Python producer drift will silently degrade Behavior Analytics. - Keep `track_id` stable across frames. Behavior Analytics derives dwell and direction from track continuity; a per-frame regenerated id makes every frame look like a new object. - Do not embed credentials, NGC tokens, or absolute customer paths in the scaffolded files. The generated tree must be portable.