Build a custom VSS SOP blueprint from the VSS 3.1 base, then deploy and test in a loop until fully operational. Use when asked to create the SOP blueprint structure, customize VSS compose for SOP, configure SOP services, set up the VSS agent for SOP, or scaffold the SOP app layer on top of met-blueprints 3.1.
Build a custom VSS SOP blueprint from the VSS 3.1 base, then deploy and test in a loop until fully operational. Use when asked to create the SOP blueprint structure, customize VSS compose for SOP, configure SOP services, set up the VSS agent for SOP, or scaffold the SOP app layer on top of met-blueprints 3.1.
owner
NVIDIA
service
vss-sop
version
1.0.0
license
CC-BY-4.0 AND Apache-2.0
reviewed
"2026-06-23T00:00:00.000Z"
metadata
{"openclaw":{"emoji":"🔧","os":["linux"]},"author":"Quan Vu <qvu@nvidia.com>","tags":["vss","sop","build","blueprint"]}
VSS SOP — Build a Custom Blueprint from VSS 3.1
Scaffold the SOP blueprint layer on top of met-blueprints 3.1. This SKILL.md is a roadmap — it points to detailed references for each stage. Read references only when working on that stage.
Overview
Use this skill when asked to create, customize, or scaffold the VSS SOP blueprint on top of standard VSS 3.1 base components.
Key scenarios:
Creating a new SOP blueprint from scratch.
Merging upstream VSS changes with custom SOP layers.
Copy NIM from upstream + apply SOP conventions (Stage 3.3) — ./agentic/vss-sop-skills/vss-sop-build/scripts/copy_nim_from_upstream.sh
Generate DS-SOP source via ds-sop-skills (RTSP output is native), then build the image (when needed) — first generate with the ds-sop-skills skill (), then run
# Prerequisites check & auto-install (run first)
./agentic/vss-sop-skills/vss-sop-deploy/scripts/preflight_check.sh --bp-repo . --fix
# Clone and prepare directories
./agentic/vss-sop-skills/vss-sop-build/scripts/clone_and_prepare.sh
# Copy references and foundations
./agentic/vss-sop-skills/vss-sop-build/scripts/copy_references.sh
./agentic/vss-sop-skills/vss-sop-build/scripts/copy_foundational_from_upstream.sh
# Modify foundational components for the SOP profile
./agentic/vss-sop-skills/vss-sop-build/scripts/modify_foundational_for_sop.sh
Verify Scaffolding Integrity
# Execute local linter and integrity verification script
./agentic/vss-sop-skills/vss-sop-build/scripts/verify_build.sh
What Changes from Upstream 3.1.0
The upstream 3.1.0 branch (video-search-and-summarization/deployments/) ships services for many blueprint variants. SOP keeps a slimmer subset and adds the sop/ app layer.
VLM_BASE_URL, VLM_NAME in sop/.env (default local DS-SOP model, customizable)
DS SOP
ds-sop:1.0.0 (local build)
ds/ds-sop/ (Stage 3.6)
Message Broker
Kafka
foundational/, topic mdx-vlm-captions
Analytics DB
Elasticsearch
foundational/, index mdx-vlm-captions-*
Pinned Container Versions
Component
Version
Elasticsearch / Kibana / Logstash
9.3.0 (stock)
VSS Agent / VST / Video Analytics API / Agent UI
3.1.0
DS-SOP
1.0.0 (local build)
Directory Structure
During build, all directories coexist under the current working directory (CWD). The upstream video-search-and-summarization/ is removed after success.
Run this first, before cloning or any build stage. It checks secret key files, GPU drivers, Docker, NVIDIA Container Toolkit, and NGC configuration, and can automatically install or configure missing prerequisites (NVIDIA Driver 580, CUDA Toolkit 13, Docker, Docker Compose, NVIDIA Container Toolkit, NGC CLI, and NGC CLI configuration).
To run the checks and automatically fix/install any missing components:
cd <bp-repo>
./agentic/vss-sop-skills/vss-sop-deploy/scripts/preflight_check.sh --bp-repo . --fix
To run the checks only, without modifying your system:
cd <bp-repo>
./agentic/vss-sop-skills/vss-sop-deploy/scripts/preflight_check.sh --bp-repo .
If any manual check fails, refer to the printed advice or the vss-sop-deploy Phase 0 references (secret keys, NVIDIA driver, Docker & Toolkit).
Stage 0.1 — Clone and Prepare
All output goes into deployments/. Upstream video-search-and-summarization/ is read-only and is cloned directly into the CWD. All paths are relative to CWD.
Step 0.1.1 — Clone Upstream Repository
Run ./agentic/vss-sop-skills/vss-sop-build/scripts/clone_and_prepare.sh to clone the upstream 3.1.0 branch of video-search-and-summarization (which serves as a read-only source for the build process).
Stage 1 — Reference → Target File Mapping
Copy these files verbatim from references/ (do NOT regenerate — they contain tuned defaults):
These references are all copied to their correct locations by running the unified copy script:
./agentic/vss-sop-skills/vss-sop-build/scripts/copy_references.sh
Transforms the upstream VSS Agent into the VSS SOP Agent (SOP compliance monitoring). No image rebuilds — done via configs, bind-mounts, env vars.
#
Step
Action
2.1
Add bp_sop_2d profile + patch volumes
Add profile to vss-ui, vss-va-mcp, vss-agent, all foundational services. Bind-mount SOP patches into vss-va-mcp. Comment out ai-agents in agents/compose.yml
DS-SOP source generation (ds-sop-skills, RTSP output native per § 18) + image build (via build_ds_sop.sh, no patch): see the Generate + Build section below + ds-sop-building.md.
Step 3.5 — Wire the SOP Compose
The compose files are wired here; both are kept in configs/:
deployments/ds/compose.yml (REPLACES upstream content) — see references/deployments/ds/compose.yml. Single include: ds-sop/ds-sop-docker-compose.yml.
Step 3.6 — Configure the DS-SOP Microservice
Copy both files verbatim from references — they encode tuned defaults, exact volume mounts, and env-var pass-through. This is done by the copy references script (./agentic/vss-sop-skills/vss-sop-build/scripts/copy_references.sh), which copies:
Full breakdown (build flow, 7 post-build checks, auto-fix recipes; RTSP is an opt-in ds-sop-skills feature, requested at generation — § 18): references/ds-sop-building.md
Build script: scripts/build_ds_sop.sh
The DS-SOP source is not cloned from the public NVIDIA SOP Inference Blueprint. It is generated by the ds-sop-skills skill into ds_sop_microservice and built as-is into the VSS blueprint's ds-sop:1.0.0 image. RTSP streaming output is an optional, opt-in ds-sop-skills feature (§ 18) that the VSS-SOP blueprint requires, so the generation prompt must request it explicitly.
Step A — Generate the source (if ds_sop_microservice does not already exist) by following the ds-sop-skills prompt — note the required"with rtsp streaming output feature" phrase:
Please follow instructions in ../ds-sop-skills/example_sop_prompt.md to generate a SOP microservice with rtsp streaming output feature in folder ../ds_sop_microservice
This emits the standard DeepStream SOP microservice (DeepStream GEBD pipeline, VLM classification, SOP checker, Kafka/JSON messaging, Basler camera support) with RTSP streaming output built in (ds-sop-skills § 18). Omitting the RTSP phrase produces a source without RTSP and the RTSP post-build checks will fail.
Step B — Build: Run ./agentic/vss-sop-skills/vss-sop-build/scripts/build_ds_sop.sh <ds_sop_microservice_dir> <bp_repo_path> (<ds_sop_microservice_dir> defaults to ../ds_sop_microservice). It validates the generated layout, ensures the binaries/ bind-mount target exists, then builds ds-sop:1.0.0. No source modification.
Notes:
RTSP is owned by ds-sop-skills (§ 18) — it is an opt-in ds-sop-skills feature, so the generation prompt must request it ("with rtsp streaming output feature"). When requested, the generated source contains the GStreamer RTSP libs + codec plugins, RTSPStreamingServer + tee1-tapped encode branch, SW_ENCODER/ENABLE_RTSP_OUTPUT/RTSP_PORT, --rtsp-port CLI, api-server auto-injection, /ds-out/{sensor_id} paths, and key-int-max=30. vss-sop-build does not add or patch any of it.
Image vs. workdir — the image is tagged ds-sop:1.0.0 at build time, but the container workdir stays /opt/nvidia/nvds_sop (as generated). The VSS ds-sop-docker-compose.yml points PYTHONPATH, working_dir, and the cache/chunks/alert mounts at /opt/nvidia/nvds_sop accordingly.
If a post-build check (§ checks in ds-sop-building.md) fails, fix it in the ds-sop-skills generation and regenerate — not by patching here.
The entire sop/ directory is new — not in the upstream repo. This is copied as part of the copy references script (./agentic/vss-sop-skills/vss-sop-build/scripts/copy_references.sh) from the skill's reference directory agentic/vss-sop-skills/vss-sop-build/references/deployments (including the top-level compose.yml, cleanup_all_datalog.sh, and the full sop/ directory with its .env file) into the target deployments/ folder.
Then customize deployments/sop/.env:
Set MDX_SAMPLE_APPS_DIR, MDX_DATA_DIR, HOST_IP, API keys (NGC_CLI_API_KEY, NVIDIA_API_KEY), HARDWARE_PROFILE.
Customize the LLM and VLM settings if needed (defaults to Llama 3.3 Nemotron Super for remote LLM and ds_sop_model for local DS-SOP VLM):
LLM Configuration (remote or local NIM): LLM_MODE, LLM_BASE_URL, LLM_NAME / LLM_NAME_SLUG.
VLM Configuration (local or remote): VLM_MODE, VLM_BASE_URL, VLM_NAME / VLM_NAME_SLUG.
All other files are used as-is.
Stage 5 — Build Verification + Deploy → Test Loop
Step 5.0 — Verify the build
Run ./agentic/vss-sop-skills/vss-sop-build/scripts/verify_build.sh — it invokes verify_build.py to check directory structure, top-level compose include, bp_sop_2d profile coverage, container image versions, Kibana dashboard flat-field mapping, SDR recorder API URLs, and SDR container cluster name syntax (checks for invalid double-double-quotes), then runs the post-verification cleanup of the upstream repo.
If a phase fails: read auto-debug output → apply fix (env, compose, patches, image rebuild) → re-deploy → re-test. Repeat until ALL phases pass with 100% check pass rate.
MANDATORY: ALL tests must pass. Do NOT classify any failure as "environmental", "intermittent", or "acceptable". Do NOT proceed with any failing test. If stuck after 3 attempts on the same failure, STOP and ask the user for help.
The pipeline is complete ONLY when the output matches:
Any partial result (e.g., 36/38) means the pipeline is NOT complete — keep debugging.
Error Handling
Build / structure issues (resolved during the build stages above):
Missing bp_sop_2d profile — must be added manually to ALL shared services in foundational/, agents/, vst/. Upstream 3.1.0 doesn't include it.
Patch mount failures — verify the Python version (python3.13) matches the VSS Agent image; mount target is /vss-agent/.venv/lib/python3.13/site-packages/vss_agents/video_analytics/.
vss-agent and vss-va-mcp containers never start with bp_sop_2d profile — both services in agents/vss-agent/vss-agent-docker-compose.yml must have bp_sop_2d in their profiles: list. The patch_profiles.py script handles this automatically. Manual fallback: add - bp_sop_2d as the first entry in the profiles: list for both vss-va-mcp and vss-agent. Also add the SOP patch volume mounts (tools.py, utils.py, es_client.py) to both services so they both use the SOP-specific video analytics module.
Video analytics API empty — vss-video-analytics-api-config.json must point to mdx-vlm-captions-* (not mdx-incidents-*).
Upstream includes fail — clean up top-level compose.yml per Stage 1 (drop developer-workflow, etc.).
sop/.env not found — must be at deployments/sop/.env (Docker Compose reads it from beside sop/compose.yml). Copied during Stage 4.
DS-SOP container won't start — confirm ds-sop:1.0.0 image exists; verify VLLM_MODEL_PATH / DDM_MODEL_PATH point to verified models.
Agent can't reach VLM — SOP uses VLM_BASE_URL=http://localhost:8300 (DS-SOP API), not a remote NIM endpoint.
Report generation fails with "error generating" / VST clip HTTP 500 ("Could not multiplex stream") — the report agent extracts a short (~2s) incident video clip via vst_video_url. With overlay_config: true, VST's OSD/bbox transcode pipeline fails to multiplex such short clips (VMSInternalError: Synchronous video generation failed). SOP incidents carry no object bounding boxes (objectId is always empty), so set overlay_config: false for the vst_video_url tool in sop/vss-agent/configs/config.yml (leave the vst_picture_url snapshot tool as-is). The plain no-overlay clip path muxes reliably.
VSS agent [401] Unauthorized calling remote LLM — NVIDIA_API_KEY='' in deployments/sop/.env is empty. deploy.sh now auto-writes the key from .secret/nvidia_build_api_key.txt into sop/.env on each run. If starting agent containers independently (not via deploy.sh), ensure NVIDIA_API_KEY is set in the shell or in sop/.env before running docker compose up.
Component-specific issues — the detailed symptoms and fixes live in the reference docs (single source of truth), so they don't drift out of sync with this skill:
Foundational / ELK pipeline — Logstash ConfigurationError at date blocks, mdx-vlm-captions index never appears, mdx-vlm-captions doc count stuck at 1 (document_id branch), Kibana "No field found for [...]" / flat-JSON data view, SOP_MESSAGING_SCHEMA/ENABLE_MESSAGING/Kafka topic issues → see foundational-building.md Troubleshooting (validated by verify_build.py --component foundational).
VIOS / VST recording & streaming — double-scheme clip URLs (http://http://...), recorder on wrong RTSP port (30556 vs 30554), "Stream not present in recorder", camera_streaming not published, SDR KeyError: 'process_type', SDR provisioning_address scheme error, SDR consumer-group round-robin, nginx host not found in upstream, vst-mcp-sopserver_portValidationError, recorder "insufficient disk capacity" → see vios-building.md Troubleshooting (validated by verify_build.py --component vios).
DS-SOP image — empty (623-byte) MP4 report clips / key-int-max keyframe interval, RTSP streaming output (ds-sop-skills § 18; opt-in at generation, built, not patched) → see ds-sop-building.md.
VSS Agent compose — vss-agent refuses to start with depends on undefined service "nvidia-nemotron-nano-9b-v2-fp8" (or -fp8-shared-gpu) → see vss-agent-building.md Step 2d.
Prerequisites & runtime issues → see vss-sop-deploy Phase 0 and Troubleshooting.
