Run post-deployment tests for the VSS SOP blueprint. Checks service health, ELK data pipeline, VIOS recording/livestream, and VSS agent (MCP, LLM, VLM, snapshot, video, report). Auto-debugs failures. Use when asked to test SOP, verify SOP deployment, check SOP services, validate SOP, run SOP health checks, or troubleshoot SOP after deploy.
Run post-deployment tests for the VSS SOP blueprint. Checks service health, ELK data pipeline, VIOS recording/livestream, and VSS agent (MCP, LLM, VLM, snapshot, video, report). Auto-debugs failures. Use when asked to test SOP, verify SOP deployment, check SOP services, validate SOP, run SOP health checks, or troubleshoot SOP after deploy.
owner
NVIDIA
service
vss-sop
version
1.0.0
license
CC-BY-4.0 AND Apache-2.0
reviewed
"2026-06-23T00:00:00.000Z"
metadata
{"openclaw":{"emoji":"🧪","os":["linux"]},"author":"Quan Vu <qvu@nvidia.com>","tags":["vss","sop","test"]}
VSS SOP — Post-Deployment Test Suite
Run this skill after the VSS SOP blueprint has been deployed (see vss-sop-deploy). It validates every layer of the stack, and when a test fails it auto-debugs by collecting container logs and printing actionable fix hints.
Overview
Verify a fresh SOP deployment is healthy
Check that ELK is receiving VLM data
Validate VIOS recording and livestream
Test the VSS agent: MCP server, LLM endpoint, VLM endpoint, snapshot, video, and report generation
Troubleshoot a broken deployment
Instructions
Quick Start
cd <bp-repo>
python agentic/vss-sop-skills/vss-sop-test/scripts/vss_sop_test.py \
--bp-repo . \
--env-file deployments/sop/.env
Run a single phase:
python agentic/vss-sop-skills/vss-sop-test/scripts/vss_sop_test.py --phase 1 # Service health only
python agentic/vss-sop-skills/vss-sop-test/scripts/vss_sop_test.py --phase 2 # ELK only
python agentic/vss-sop-skills/vss-sop-test/scripts/vss_sop_test.py --phase 3 # VIOS only
python agentic/vss-sop-skills/vss-sop-test/scripts/vss_sop_test.py --phase 4 # VSS Agent only
Test Phases
Phase 1 — Service Health
Checks that all required Docker containers are running (Up).
Manual fix: Redeploy with docker compose -f compose.yml --env-file sop/.env --profile bp_sop_2d up -d.
Phase 2 — ELK Data Pipeline
Elasticsearch reachable — GET localhost:9200/_cluster/health
Cluster health — must be green or yellow
Indices exist — at least one mdx-vlm-captions-* index
VLM messages — mdx-vlm-captions-* index has > 0 documents
Kibana dashboard fields — verifies the actual ES mapping has the flat JSON fields the dashboard expects (response, sensor_id, cv_execute_time, vlm_execute_time, chunk_idx, frame_number, @timestamp), and that no protobuf-style nested fields (llm, sensor) are present instead
Kibana ndjson fields — queries Kibana's saved objects API to confirm the imported data view's runtime fields reference the correct flat field names (not llm.queries.response.keyword or sensor.id.keyword), and that timeFieldName is @timestamp
Auto-debug: Tails mdx-logstash, mdx-elastic, and mdx-kibana logs.
# Check the actual field structure in Elasticsearch
curl localhost:9200/mdx-vlm-captions-*/_mapping?pretty
# If fields are flat (response, sensor_id) but ndjson uses nested (llm.queries.response):# Re-import the corrected ndjson
curl -X POST localhost:5601/api/saved_objects/_import?overwrite=true \
-H 'kbn-xsrf: true' \
--form file=@deployments/sop/sop-app/kibana-dashboard/sop-kibana-objects.ndjson
Phase 3 — VIOS Recording & Livestream
VST reachable — GET localhost:30888/vst/api/v1/sensor/list
Sensors registered — at least one sensor (typically sensor_0)
Streams available — /sensor/streams returns data
Recording — checks /record/status per sensor
Livestream — checks /live/streams per sensor
Auto-debug: Tails sensor-ms-sop, rtspserver-ms-1-sop, recorder-ms-1-sop, and sdr-http-recorder-sop logs.
Manual fix if no sensor:
cd <bp-repo>/deployments/sop/sop-app/helper-scripts
nohup ./run_rtsp_test.sh > client.log &
Phase 4 — VSS Agent End-to-End
#
Test
Method
4.1
MCP health
GET localhost:9901/health
4.2
LLM endpoint
GET <LLM_BASE_URL>/v1/models
4.3
VLM endpoint
GET <VLM_BASE_URL>/v1/models
4.4
Agent health
GET localhost:8000/health
4.5
Snapshot from VIOS
Agent chat: "Take a snapshot of sensor sensor_0"
4.6
Video → VLM
Agent chat: "What is the current SOP status for sensor sensor_0?"
4.7
Report generation
Agent chat: "Generate an SOP compliance report for sensor sensor_0 from … to …"
Auto-debug: Tails vss-agent, vss-va-mcp, and mdx-ds-sop-1 logs for any failing sub-test.
Phase 4.7 "no streams found" on fresh deployment:
If the test runs immediately after deploy, the report time window (now-1min to now) may
precede the first recording. Wait at least 1 minute after the RTSP stream registers before running
Phase 4.
Examples
Pass Criteria — MANDATORY
ALL tests must pass (100%). There is NO acceptable failure rate.
Rules the agent MUST follow:
Zero tolerance for failures. Every single test check across all phases MUST pass. A result of 36/38 or 17/19 is a FAIL — not a pass with caveats.
Do NOT classify failures as "environmental", "infrastructure", "test harness", or "intermittent". Every failure is a bug until you have fixed it and the test passes on re-run.
Do NOT lower the pass threshold. Never say "good enough", "residual", "non-blocking", or "acceptable failure". If a test fails, fix it.
Do NOT skip or move on. If a test fails after 3 fix attempts, STOP and ask the user for help. Do NOT proceed to the next phase or declare completion.
Do NOT mark a phase as PASS if any check within it failed. Use FAIL until every check is green.
Timeout failures are bugs too. If a test times out (e.g., 30s client read-timeout), increase the timeout or fix the underlying latency — do NOT dismiss it as a test configuration issue.
Re-run after every fix. After applying a fix, re-run the failing phase to confirm the fix works. A fix is not verified until the test passes.
Completion criteria: The pipeline is complete ONLY when the final report shows 38/38 (or equivalent total) with zero failures across all phases. Any other result means more work is needed.
Workflow for the Agent
Copy this checklist and track progress:
Test Progress:
- [ ] Phase 1: Service health — ALL checks PASS
- [ ] Phase 2: ELK data pipeline — ALL checks PASS
- [ ] Phase 3: VIOS recording & livestream — ALL checks PASS (7/7)
- [ ] Phase 4: VSS agent (MCP, LLM, VLM, snapshot, video, report) — ALL checks PASS (7/7)
- [ ] Auto-debug any failures (do NOT skip — every failure must be fixed)
- [ ] Auto-update vss-sop-build skills/scripts if root cause found
- [ ] Confirm 100% pass rate before reporting to user
- [ ] Report summary to user
Step 1 — Determine <bp-repo>
# Usually the vss-sop checkout
BP_REPO=$(pwd) # or wherever the user cloned vss-sop
If a test still fails after 3 fix attempts, STOP and ask the user for help. Do NOT declare completion or proceed to the next step.
Never rationalize a failure. Statements like "this is an environmental issue, not a code defect" are NOT acceptable reasons to skip a fix. Either fix the environment or fix the test.
Auto-Updating the Upstream Skill After Root-Cause Fix
When a test failure is debugged and the root cause is identified, update the corresponding upstream skill so the bug cannot recur. First determine where the root cause lives, then update the correct skill:
Keep every change simple and general — fix the pattern, not just the instance.
4a. Root cause in build/deploy → update vss-sop-build
Root-cause category
What to update in vss-sop-build
Missing/wrong config value (.env, YAML)
Fix the default in the matching reference file under agentic/vss-sop-skills/vss-sop-build/references/
Missing build step or wrong ordering
Add/reorder step in the relevant *-building.md reference doc
Compose issue (profile, volume, depends_on)
Patch the reference compose file(s) under agentic/vss-sop-skills/vss-sop-build/references/deployments/ or agentic/vss-sop-skills/vss-sop-build/references/configs/
Script bug (copy, patch, verify)
Fix the script under agentic/vss-sop-skills/vss-sop-build/scripts/
Undocumented failure mode
Add entry to vss-sop-build SKILL.md ## Error Handling section
4b. Root cause in DS-SOP source code → update ds-sop-skills
Root-cause category
What to update in ds-sop-skills
API endpoint / schema bug
Fix the reference in ../ds-sop-skills/deepstream-sop/references/skill_01_fastapi_endpoints.md or ../ds-sop-skills/deepstream-sop/references/skill_02_pydantic_schemas.md
DeepStream pipeline issue
Fix the reference in ../ds-sop-skills/deepstream-sop/references/skill_03_deepstream_pipeline.md or ../ds-sop-skills/deepstream-sop/references/skill_04_config_templates.md
Triton / DDM model issue
Fix the reference in ../ds-sop-skills/deepstream-sop/references/skill_05_triton_ddm_model.md or ../ds-sop-skills/deepstream-sop/references/skill_05b_custom_postprocess.md
SOP process / checker logic
Fix the reference in ../ds-sop-skills/deepstream-sop/references/skill_06_sop_process_manager.md or ../ds-sop-skills/deepstream-sop/references/skill_06b_sop_checker.md
VLM inference bug
Fix ../ds-sop-skills/deepstream-sop/references/vllm_inference_reference.py or the corresponding skill doc
RTSP streaming output
Fix ../ds-sop-skills/deepstream-sop/references/skill_18_rtsp_streaming_output.md or the reference code
Messaging / Kafka schema
Fix ../ds-sop-skills/deepstream-sop/references/skill_16_message_schema.md or ../ds-sop-skills/deepstream-sop/references/messager_reference.py
Dockerfile / build
Fix ../ds-sop-skills/deepstream-sop/references/Dockerfile_reference or ../ds-sop-skills/deepstream-sop/references/skill_09_docker_build_deploy.md
Reference source file bug
Fix the corresponding *_reference.py / *_reference.cpp file under ../ds-sop-skills/deepstream-sop/references/
Undocumented failure mode
Add entry to ../ds-sop-skills/deepstream-sop/SKILL.md error handling or the relevant skill doc
Procedure (applies to both 4a and 4b):
Classify the root cause — determine whether it belongs to build/deploy config (vss-sop-build) or DS-SOP source code (ds-sop-skills).
Identify the file — map the root cause to the corresponding source file in the chosen skill's directory.
Make the minimal fix — change only what is needed; prefer updating a default value, adding a guard/check, or appending an entry under Error Handling over larger rewrites.
Verify consistency:
If a vss-sop-build reference was changed, re-run agentic/vss-sop-skills/vss-sop-build/scripts/verify_build.sh to confirm the build still passes.
If a ds-sop-skills reference was changed, regenerate the source and rebuild ds-sop:1.0.0 to confirm the fix.
Log the change — include in the final report which skill and file was updated and why.
Rule of thumb: if the root cause is in the generated DS-SOP source code (Python logic, pipeline config, API, Dockerfile, etc.), the fix belongs in ds-sop-skills — so the next generation produces correct code. If it is in the VSS blueprint build process (compose wiring, .env defaults, copy scripts, build ordering), the fix belongs in vss-sop-build. If it is environment-specific (wrong IP, expired key), just document it in the report — don't change either skill.
Step 5 — Report results
Present the final summary table to the user. The report MUST show 100% pass rate across all phases.
If all tests pass: report the green summary.
If any test still fails: report the failure with full debug context and explicitly state that the pipeline is NOT complete. Do NOT use language suggesting partial success (e.g., "36/38 checks pass" is a failure report, not a success report).
If any vss-sop-build or ds-sop-skills files were updated in Step 4, list them with a one-line summary of each change.
Service Endpoints Reference
Service
URL
Elasticsearch
http://localhost:9200
Kibana
http://<EXTERNAL_IP>:5601
VST (VIOS)
http://localhost:30888/vst
VSS Agent
http://localhost:8000
VSS VA MCP
http://localhost:9901
LLM NIM
Configured via LLM_BASE_URL in .env
VLM NIM
Configured via VLM_BASE_URL in .env
VSS UI
http://<EXTERNAL_IP>:3000
Phoenix
http://<EXTERNAL_IP>:6006
Dependencies
The test script requires only requests (installed in most Python environments). No additional pip install needed if you have a standard Python 3.8+ environment.
