Orchestrate the end-to-end SOP pipeline, including preflight prerequisite checks, verifying models and downloading assets, generating the DeepStream SOP microservice with RTSP output, evaluating the microservice, and building, deploying, and testing the VSS SOP blueprint. Use when asked to run the full SOP pipeline, set up the SOP pipeline from scratch, execute preflight checks, verify models, download assets, generate the SOP microservice, evaluate the microservice, build the VSS blueprint, deploy the VSS blueprint, test the VSS blueprint, or manage the complete build-evaluate-deploy-test cycle.
Orchestrate the end-to-end SOP pipeline, including preflight prerequisite checks, verifying models and downloading assets, generating the DeepStream SOP microservice with RTSP output, evaluating the microservice, and building, deploying, and testing the VSS SOP blueprint. Use when asked to run the full SOP pipeline, set up the SOP pipeline from scratch, execute preflight checks, verify models, download assets, generate the SOP microservice, evaluate the microservice, build the VSS blueprint, deploy the VSS blueprint, test the VSS blueprint, or manage the complete build-evaluate-deploy-test cycle.
owner
NVIDIA
service
vss-sop
version
1.0.0
license
CC-BY-4.0 AND Apache-2.0
reviewed
"2026-06-23T00:00:00.000Z"
metadata
{"openclaw":{"emoji":"🚀","os":["linux"]},"author":"Quan Vu <qvu@nvidia.com>","tags":["sop","build","pipeline"]}
Full SOP Pipeline — Build, Evaluate, Deploy, Test
Orchestrates the complete SOP pipeline from asset download through deployment validation. Each phase must succeed before proceeding to the next.
Overview
Set up the entire SOP pipeline from scratch on a new machine
Re-run the full pipeline after upstream changes
Rebuild and redeploy after ds-sop-skills or vss-sop-build changes
Prerequisites
Requirement
Detail
GPU
NVIDIA H100 / H200 / A100 with >= 80 GB VRAM
Driver
NVIDIA 580+ with CUDA 13
Software
Docker with BuildKit, NVIDIA Container Toolkit 1.18.1, ngc CLI
Secret key
<bp-repo>/.secret/ngc_api_key.txt must contain a valid NGC API key
ds-sop-skills
../ds-sop-skills/ directory with deepstream-sop/SKILL.md and prompt files
Usage
Phase
What
Key Script / Skill
Pre
Prerequisites Check & Auto-Install (run first)
vss-sop-deploy/scripts/preflight_check.sh --fix
0
Verify Models & Download Assets
vss-sop-deploy/scripts/download_assets.sh
1
Generate SOP Microservice
../ds-sop-skills/example_sop_prompt.md
2
Evaluate SOP Microservice
../ds-sop-skills/eval_sop_prompt.md
3
Build VSS-SOP Blueprint
vss-sop-build skill
4
Deploy VSS-SOP Blueprint
vss-sop-deploy skill
5
Test VSS-SOP Blueprint
vss-sop-test skill
Phase Pre — Prerequisites Check & Auto-Install
Run this first, before any other phase. It checks secret key files, GPU drivers, Docker, NVIDIA Container Toolkit, and NGC configuration, and can automatically install or configure missing prerequisites (NVIDIA Driver 580, CUDA Toolkit 13, Docker, Docker Compose, NVIDIA Container Toolkit, NGC CLI, and NGC CLI configuration).
cd <bp-repo>
# Run checks and auto-fix/install any missing components
./agentic/vss-sop-skills/vss-sop-deploy/scripts/preflight_check.sh --bp-repo . --fix
To run the checks only, without modifying your system:
cd <bp-repo>
./agentic/vss-sop-skills/vss-sop-deploy/scripts/preflight_check.sh --bp-repo .
Verify: the script reports all prerequisites satisfied (secret keys, NVIDIA driver, Docker, NVIDIA Container Toolkit, NGC CLI & config). If any manual check fails, resolve it per the printed advice before proceeding. See vss-sop-deploy Phase 0 for reference docs.
Phase 0 — Verify Models & Download Assets
For optimal accuracy, you must retrain/fine-tune the models, which can be done using the SOP Training Blueprint. After training, move model and config to /opt/models/... and /opt/sop/... directories.
cd <bp-repo>
./agentic/vss-sop-skills/vss-sop-deploy/scripts/download_assets.sh --bp-repo .
This performs:
Checks that the required trained model and config files exist under /opt/models and /opt/sop. If not, prompts the user to retrain/fine-tune the models using the SOP Training Blueprint.
Downloads and re-encodes the sample RTSP video to H.264 at 30 FPS.
Sets up the ds-sop cache directory and data log directories.
Verify:/opt/models/vlm/checkpoint/ contains VLM weights, /opt/models/gbed_models/ddm/checkpoint.pth.tar exists, /opt/sop/configs/actions.json and /opt/sop/configs/vlm_prompts.txt exist.
Phase 1 — Generate SOP Microservice
Generate the DeepStream SOP microservice with RTSP streaming output into ../ds_sop_microservice. Skip if ../ds_sop_microservice/ already exists with the expected layout.
The generation prompt must include the RTSP streaming output feature request (../ds-sop-skills § 18), which the VSS-SOP blueprint requires:
Please follow instructions in ../ds-sop-skills/example_sop_prompt.md to generate a SOP microservice with rtsp streaming output feature ($18) in folder ../ds_sop_microservice
This follows ../ds-sop-skills/deepstream-sop/SKILL.md to generate:
FastAPI microservice with /v1/chat/completions SSE endpoint
DeepStream GEBD pipeline with DDM-Net via Triton CAPI
VLM inference (Cosmos Reason) for action classification
Run the vss-sop-deploy skill from the blueprint repo root:
cd <bp-repo>
# Follow vss-sop-deploy SKILL.md Phases 0–5
Deployment phases:
Phase 0: Prerequisites check and auto-install (preflight_check.sh --bp-repo . --fix) — already run in Phase Pre above; re-runs as an idempotent safety check.
Phase 1: Download models & assets (already done in Phase 0 above — will skip if present).
Phase 2: Start RTSP server simulation (start_rtsp_server.sh --bp-repo .).
Phase 3: Build DS-SOP Docker image & verify components (already built in Phase 3 above — verify only).
Phase 4: Configure & deploy the blueprint (configure_blueprint.sh, deploy.sh).
Phase 5: Test with RTSP stream (test_rtsp.sh --bp-repo .).
Apply the fix (env, compose, patches, image rebuild, source regeneration).
Re-run the failing phase to confirm the fix.
Repeat until all checks in that phase pass.
If the root cause is in the generated DS-SOP source code, update ds-sop-skills references and regenerate. If the root cause is in the build/deploy config, update vss-sop-build references.
Examples
After all phases complete with 100% pass rate, present a summary table:
Category
Status
Details
Phase Pre: Prerequisites
Pass/Fail
Driver, Docker, Toolkit, NGC config
Phase 0: Assets
Pass/Fail
Models and configs at /opt
Phase 1: Generate
Pass/Fail
../ds_sop_microservice with RTSP
Phase 2: Evaluate
Pass/Fail
Unit tests + API evaluation
Phase 3: Build
Pass/Fail
Per-stage build status
Phase 4: Deploy
Pass/Fail
Per-phase deploy status
Phase 5: Test
Pass/Fail
Per-phase test status (must be 100%)
Include:
Any vss-sop-build or ds-sop-skills files updated during debugging.
Service endpoints: VSS-UI, Kibana, VIOS-UI, Grafana.
If any test is still failing, the report must clearly state "Pipeline Incomplete — X failures remain" and list them. Do not present a partial success as completion.
