nvidia-video-search-and-summarization-alerts

name	nvidia-video-search-and-summarization-alerts
description	Manage and monitor VSS alerts after the alerts profile is deployed. The deployment's mode (CV vs VLM real-time) is fixed at deploy time and determines the workflow — start/stop real-time alerts via the VSS Agent on a VLM deployment, onboard CV alerts by adding RTSP streams to VIOS on a CV deployment, query incidents, customize verifier prompts. Use when asked to start/stop a real-time alert, check or list alerts, add a camera, use a sample video for alerts, customize alert prompts, or view verdicts.
license	Apache-2.0
metadata	{"version":"3.1.0","github-url":"https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization","tags":"nvidia blueprint operational","author":"NVIDIA"}

VSS Alert Management

The alerts profile is deployed in one of two modes at a time. The mode is chosen at deploy -p alerts -m {verification,real-time} and is static until you tear down and redeploy. Which mode is running determines which flow to use — this skill does not route per-request.

When to Use

• Start or stop a real-time alert on a sensor ("Start real-time alert for boxes dropped on sensor warehouse_sample")
• List or query detected incidents / alerts
• Add a new camera to the alerts pipeline
• Customize the VLM-verifier prompts (CV mode)
• Check verdicts (confirmed / rejected / unverified)

---

Deployment prerequisite

This skill requires the VSS alerts profile running on the host at $HOST_IP, in either verification or real-time mode. Before any request:

1. Probe the stack:

   # Either perception-alerts (CV mode) OR rtvi-vlm (VLM mode) must be present.
   curl -sf --max-time 5 "http://${HOST_IP}:8000/docs" >/dev/null \
     && docker ps --format '{{.Names}}' \
        | grep -qE '^(perception-alerts|rtvi-vlm)$'
   

2. If the probe fails, ask the user:

   "The VSS alerts profile isn't running on $HOST_IP. Which mode should I deploy — verification (CV) or real-time (VLM)?"

   • Answer → hand off to the /deploy skill with -p alerts -m <mode>. Return here once it succeeds.
   • If the user declines → stop. Do not run this skill against a missing stack.

   (If your caller has granted explicit pre-authorization to deploy autonomously — e.g. the request says "pre-authorized to deploy prerequisites", or you are running in a non-interactive evaluation harness with that permission — skip the confirmation and invoke /deploy directly. Default the mode to verification unless the request specifies otherwise.)

3. If the probe passes, detect the mode per § Step 1 below.

---

The Two Modes (Deploy-Time Choice)

Mode	Deploy flag	Env (.env)	What runs	How alerts are created
CV (verification)	-m verification	MODE=2d_cv	RT-CV (Grounding DINO) + Behavior Analytics + alert-bridge VLM verifier	Static: any RTSP flowing through VIOS is auto-processed; Behavior Analytics emits candidates; VLM verifies each clip per alert_type_config.json
VLM (real-time)	-m real-time	MODE=2d_vlm	rtvi-vlm continuous inference	Dynamic: user asks the VSS Agent to start monitoring a sensor with a natural-language detection prompt

Mode is static. Switching requires deploy down + deploy up with the other -m flag — see the deploy skill. Never assume both flows are available at once.

---

Step 1 — Detect the Currently Deployed Mode

Before running any alert workflow, check which mode is live. Use container names as the signal:

# VLM real-time mode
docker ps --format '{{.Names}}' | grep -qx rtvi-vlm && echo "mode=VLM"

# CV verification mode (behavior analytics is CV-only; alert-bridge is the VLM verifier)
docker ps --format '{{.Names}}' | grep -qx vss-behavior-analytics-alerts && echo "mode=CV"

Exactly one of these should match on a healthy alerts deployment. If neither matches, the alerts profile is not deployed — direct the user to the deploy skill.

Alternative signal (if docker ps isn't available in the current context): check the profile's .env:

grep -E '^MODE=' deployments/developer-workflow/dev-profile-alerts/.env
# MODE=2d_cv   → CV mode
# MODE=2d_vlm  → VLM real-time mode

---

Step 2 — Route by Deployed Mode

Deployed mode	User asks about…	Action
CV	any alert request	Run Workflow A (CV) — onboard RTSP via vios skill, then query alerts. No per-request create call.
CV	specifically a VLM real-time alert ("start alert for boxes dropped…")	Redeployment required. Confirm with the user first, then point to the deploy skill to tear down and redeploy with -m real-time. Do not attempt to run the VLM flow on a CV deployment — the agent's rtvi_vlm_alert tool will fail because rtvi-vlm is not running.
VLM	any alert request	Run Workflow B (VLM) — call the VSS Agent with a detection prompt.
VLM	specifically a CV / behavior-analytics / PPE-rule alert	Redeployment required. Confirm, then point to deploy skill for -m verification. The CV pipeline (RT-CV, Behavior Analytics, alert-bridge) is not running on a VLM deployment.

Always confirm before triggering a redeploy. A mode switch stops all currently-running monitoring and restarts services.

---

Prereq for Either Mode: Sensor Must Be in VIOS

Both modes require the camera to be registered in VIOS first.

• If the user hands you only an RTSP URL (or an IP camera) — defer to the vios skill to add it via POST /sensor/add (see vios skill Section 6). Record the returned sensorId / name.
• If the user names an existing sensor — confirm it is listed by GET /sensor/list via the vios skill before proceeding.
• If the user asks to use a local/sample MP4 for a VLM real-time alert, do not upload the MP4 directly to VIOS storage (PUT /storage/file/...). VIOS file uploads create sensor_file entries whose stream URL is a local file path; rtvi-vlm requires a live rtsp://... URL.
• For a local/sample MP4 in VLM real-time mode, first add the video to NVStreamer (or another RTSP restreamer) and obtain its RTSP live URL, then add that RTSP URL to VIOS via POST /sensor/add with the desired sensor name. Only proceed once GET /sensor/<sensorId>/streams shows a stream URL starting with rtsp://.

On a CV deployment, adding the RTSP is the entire onboarding step — the pipeline picks up the stream automatically once it is in VIOS. On a VLM deployment, adding the RTSP is a prerequisite to Workflow B.

---

The Agent /generate Endpoint

All VLM-flow actions and all query actions go through the VSS Agent's natural-language endpoint:

AGENT="http://<AGENT_ENDPOINT>"   # default http://localhost:8000 on the alerts profile

curl -s -X POST "$AGENT/generate" \
  -H "Content-Type: application/json" \
  -d '{"input_message": "<natural-language request>"}' | jq .

Endpoint resolution: use the agent endpoint from the active VSS deployment context. If unavailable, ask the user. Do not discover via filesystem.

Availability check: curl -sf --connect-timeout 5 "$AGENT/docs".

Do not call the rtvi-vlm microservice endpoints directly — always go through the agent. The agent internally dispatches to rtvi_vlm_alert, rtvi_prompt_gen, and video_analytics_mcp.get_incidents.

---

Workflow A — CV Mode (deployment is -m verification / MODE=2d_cv)

On a CV deployment, alerts are deployment-driven, not request-driven. There is no agent call to "create" an alert.

1. Onboard the camera — add the RTSP to VIOS via the vios skill (POST /sensor/add). Once registered and online, the CV pipeline picks it up automatically.

2. Confirm the sensor is online:

   curl -s "http://<VST_ENDPOINT>/vst/api/v1/sensor/<sensorId>/status" | jq .
   

3. Wait for alerts to land in Elasticsearch. Behavior Analytics emits candidates that match configured rules; alert-bridge calls the VLM to confirm/reject each candidate per alert_type_config.json. Use Workflow C to query results.

If the user asks you to "start a real-time alert" on a CV deployment, that is a mode mismatch — see the routing table above.

---

Workflow B — VLM Mode (deployment is -m real-time / MODE=2d_vlm)

On a VLM deployment, the user drives alert creation via natural-language requests to the VSS Agent. The agent calls rtvi_prompt_gen to turn the description into a Yes/No detection question, then rtvi_vlm_alert with action="start" to register the stream with rtvi-vlm and begin continuous monitoring.

Before calling the agent, verify the target sensor is RTSP-backed:

curl -s "http://<VST_ENDPOINT>/vst/api/v1/sensor/<sensorId>/streams" | jq .

At least one stream for the sensor must have a URL beginning with rtsp://. If the stream URL is a local file path such as /home/vst/.../video.mp4, the sensor was uploaded as a VIOS file and real-time alert start will fail. For sample videos, add the video to NVStreamer first, register the returned RTSP URL in VIOS, then start the alert against that RTSP-backed VIOS sensor.

Sample-video onboarding for real-time alerts:

1. Add the MP4 to NVStreamer and get the RTSP live URL for the new stream.

2. Register that RTSP URL in VIOS:

   curl -s -X POST "http://<VST_ENDPOINT>/vst/api/v1/sensor/add" \
     -H "Content-Type: application/json" \
     -d '{
       "sensorUrl": "rtsp://<nvstreamer-host>:<port>/<path>",
       "name": "warehouse_sample"
     }' | jq .
   

3. Confirm GET /sensor/warehouse_sample/streams returns the RTSP URL, then call the VSS Agent as shown below.

Canonical sample request:

curl -s -X POST "$AGENT/generate" \
  -H "Content-Type: application/json" \
  -d '{"input_message": "Start real-time alert for boxes dropped on sensor warehouse_sample"}' | jq .

More examples:

# Vehicle collisions on a street cam
curl -s -X POST "$AGENT/generate" -H "Content-Type: application/json" \
  -d '{"input_message": "Start real-time alert for vehicle collisions on sensor Camera_02"}' | jq .

# Forklift-pedestrian proximity
curl -s -X POST "$AGENT/generate" -H "Content-Type: application/json" \
  -d '{"input_message": "Monitor Warehouse_Dock_3 for a forklift passing within 1 meter of a pedestrian"}' | jq .

# Generic start (no specific target — uses default prompt)
curl -s -X POST "$AGENT/generate" -H "Content-Type: application/json" \
  -d '{"input_message": "Start real-time alert for sensor warehouse_sample"}' | jq .

What the agent does under the hood:

1. rtvi_prompt_gen — converts "boxes dropped" → prompt: "Detect for a box being dropped. Answer in Yes or No", system_prompt: "You are a helpful assistant.".
2. rtvi_vlm_alert action="start" — looks up the sensor in VIOS live streams, calls POST /v1/streams/add and POST /v1/generate_captions_alerts (stream=true) on rtvi-vlm. Returns stream_id.

Alert semantics: every chunk is captioned; a chunk whose VLM response contains "yes" or "true" (case-insensitive) triggers an incident published to the Kafka incident topic (mdx-vlm-incidents on the alerts profile). That is why prompts must force a Yes/No answer.

Stop monitoring:

curl -s -X POST "$AGENT/generate" -H "Content-Type: application/json" \
  -d '{"input_message": "Stop real-time alert for sensor warehouse_sample"}' | jq .

If the user asks you to flag a scenario that matches a CV alert_type (e.g. "ladder PPE violation") on a VLM deployment, that is a mode mismatch — see the routing table above.

---

Workflow C — Query / List Alerts (works on either mode)

Both CV- and VLM-generated alerts land in Elasticsearch and are queryable via the agent's video_analytics_mcp.get_incidents tool.

curl -s -X POST "$AGENT/generate" -H "Content-Type: application/json" \
  -d '{"input_message": "Show me recent alerts for sensor warehouse_sample"}' | jq .

curl -s -X POST "$AGENT/generate" -H "Content-Type: application/json" \
  -d '{"input_message": "List confirmed alerts from the last hour"}' | jq .

curl -s -X POST "$AGENT/generate" -H "Content-Type: application/json" \
  -d '{"input_message": "Were there any PPE violations today on Camera_02?"}' | jq .

curl -s -X POST "$AGENT/generate" -H "Content-Type: application/json" \
  -d '{"input_message": "Show collision incidents from Camera_02 between 2026-04-23T00:00:00.000Z and 2026-04-23T23:59:59.000Z"}' | jq .

For richer / non-natural-language filtering (sensor-level, time-series, counts): use the video-analytics skill (VA-MCP on port 9901).

Verdict interpretation (CV mode only)

Verified alerts carry an extended info block:

verdict	Meaning
confirmed	VLM determined the alert is real
rejected	VLM determined it is a false positive
unverified	Verification could not complete (error)

Check verification_response_code (200 = success) and reasoning for the VLM's explanation. VLM-mode incidents are always "confirmed" at source (the trigger itself is a Yes/No VLM answer), so there is no separate verdict field.

---

Customize CV Verifier Prompts (CV mode only)

CV-path verifier prompts live in:

deployments/developer-workflow/dev-profile-alerts/vlm-as-verifier/configs/alert_type_config.json

Each entry maps a CV alert_type (the category field emitted by Behavior Analytics) to the VLM prompts used for verification:

{
  "version": "1.0",
  "alerts": [
    {
      "alert_type": "FOV Count Violation",
      "output_category": "Ladder PPE Violation",
      "prompts": {
        "system": "You are a helpful assistant.",
        "user": "Is anyone on the ladder without a hardhat and safety vest? Answer yes or no.",
        "enrichment": "Describe the PPE violation in detail..."
      }
    }
  ]
}

• alert_type must match the category emitted by Behavior Analytics.
• output_category is the display name in Elasticsearch / UI.
• enrichment (optional) triggers a second VLM call for a richer description; requires alert_agent.enrichment.enabled: true.
• Changes require a restart of the alert-bridge (vlm-as-verifier) container.

VLM real-time prompts are not configured in a file — they are per-request, shaped by rtvi_prompt_gen from the user's natural-language detection description.

---

Cross-Skill Links

Task	Skill
Deploy, redeploy, or switch alert mode	deploy skill — deploy -p alerts -m {verification,real-time}
Add an RTSP / IP camera to VIOS	vios skill — Section 6 (Add Sensor / Stream)
List sensors, take a snapshot, download a clip	vios skill
Time-range incident / occupancy / PPE metrics from Elasticsearch	video-analytics skill (VA-MCP :9901)
Generate a detailed incident report from an alert	incident-report skill

---

Gotchas

• Mode is static. Do not attempt to run the VLM flow on a CV deployment or vice versa — required services won't be running. Confirm with the user, then route to the deploy skill for redeployment.
• A mode switch tears down the current deployment. Any running VLM monitoring streams and any CV alert state not already in Elasticsearch will be lost.
• Don't call the rtvi-vlm microservice directly from this skill. Always go through $AGENT/generate. The agent handles sensor→RTSP lookup, stream registration, and teardown.
• Sensor must already be in VIOS for either mode. If the user hands you only an RTSP URL, use the vios skill first.
• VLM alert trigger is a "yes" / "true" token match on the VLM response (case-insensitive). rtvi_prompt_gen enforces the Yes/No pattern — don't hand-craft prompts that break it.
• Stopping a VLM alert is one agent call ("Stop real-time alert…"); the agent handles both the caption-stream and the stream-registration teardown.
• Prompt changes to alert_type_config.json need an alert-bridge restart. alert_agent.enrichment.enabled: true is required for the enrichment prompt to fire.