vss-manage-alerts

## Purpose Operate the VSS alert pipeline (mode detection, Alert-Bridge subscriptions, verification verdicts, on-demand verification, always-on operation, Slack notifications, queries, camera onboarding, verifier-prompt customization). ## Prerequisites - Active VSS **alerts** profile reachable either on Docker (`$HOST_IP:9080` Alert Bridge) or through the public Ingress (`VSS_PUBLIC_URL` with `/alert-bridge`). - Follow [`../vss-build-vision-ai/references/deployment_resolution.md`](../../vss-build-vision-ai/references/deployment_resolution.md) for the shared `VSS_PUBLIC_URL` contract. - `curl` and `jq` on the agent host. Docker Compose mode detection may use `docker` / `generated.env`; Kubernetes must not. ## Instructions Follow the routing tables and step-by-step workflows below. Each section that ends in *workflow*, *quick start*, or *flow* is intended to be executed top-to-bottom. Detailed reference material lives in `references/` and helper scripts live in `scripts/` — call them via `run_script` when the skill points to a script by name. ## Examples Runnable end-to-end scenarios live under `evals/` (each `*.json` manifest); inline `curl` blocks appear in each workflow below. Replay with `nv-base validate <this-skill-dir> --agent-eval`. ## Limitations Requires the matching VSS profile/microservice deployed and reachable. NGC-hosted models/NIMs are subject to rate-limits, GPU-memory needs, and license terms; concurrency and storage limits depend on host hardware and the profile's compose file. ## Troubleshooting - **Connection refused** → microservice not running: probe `/docs` or `/health`, redeploy via `vss-build-vision-ai`. - **HTTP 401/403 on NGC pulls** → missing/expired `NGC_CLI_API_KEY`: `docker login nvcr.io` and re-export the key. - **OOM / model load failure** → insufficient GPU memory: use a smaller variant or `docker compose down` to free GPUs. # VSS Alert Management The alerts profile runs in one of two modes (chosen at the `/vss-build-vision-ai` stock Alerts workflow in verification or real-time mode) — see **The Two Modes** table below. This skill routes by **deployed mode + user intent** (monitoring vs subscription CRUD vs Slack webhook), driving the **Alert Bridge REST API directly** (no VSS Agent `/generate`). ## When to Use - Start/stop a real-time alert on a sensor ("Start real-time alert for boxes dropped on warehouse_sample") - Create/list/stop realtime subscription rules on Alert Bridge - Set up or manage Slack incident notifications - List or query detected incidents / alerts (Workflow C) - Inspect CV verification results and verdicts (confirmed/rejected/not-confirmed/verification-failed), explain how verification works, customize VLM-verifier prompts (CV mode — Workflow B) - Run a one-shot on-demand verification of a specific video/image URL (CV mode — Workflow F) - Check whether always-on alerting is active, query its incidents, troubleshoot missing always-on alerts (VLM real-time — Workflow G; operate only, no config authoring) - Add a new camera to the alerts pipeline (Workflow A) --- ## Deployment prerequisite Requires the VSS **alerts** profile in either `verification` (CV) or `real-time` (VLM) mode. Resolve endpoints once before probing. See [`../vss-build-vision-ai/references/deployment_resolution.md`](../../vss-build-vision-ai/references/deployment_resolution.md). ```bash # Prefer VSS_PUBLIC_URL; accept legacy VSS_ENDPOINT as the same public origin. if [ -z "${VSS_PUBLIC_URL:-}" ] && [ -n "${VSS_ENDPOINT:-}" ]; then VSS_PUBLIC_URL="${VSS_ENDPOINT}" fi if [ -n "${VSS_PUBLIC_URL:-}" ]; then DEPLOYMENT_KIND="kubernetes" VSS_PUBLIC_URL="${VSS_PUBLIC_URL%/}" # Force public prefixes — ignore leftover Docker AB / VST / VA_MCP host ports. AB="${VSS_PUBLIC_URL}/alert-bridge" VA_MCP_URL="${VSS_PUBLIC_URL}/va-mcp" else DEPLOYMENT_KIND="docker" : "${HOST_IP:?Set HOST_IP for Docker Compose or VSS_PUBLIC_URL for Kubernetes}" AB="http://${HOST_IP}:9080" VA_MCP_URL="http://${HOST_IP}:9901" fi ``` On Kubernetes, do not use `kubectl port-forward`, Service DNS, NodePorts, `docker exec`, `docker inspect`, or `docker ps`. Probe Alert Bridge with `curl -sf --max-time 5 "$AB/health"` (`/health`, not `/api/v1/health`). ```bash # Reachability — either mode. Alert Bridge must answer before operate work. curl -sf --max-time 5 "$AB/health" >/dev/null ``` Optional Docker-only peer check (skipped on Kubernetes — RT-VLM is not on alerts Ingress): ```bash if [ "${DEPLOYMENT_KIND:-docker}" != "kubernetes" ]; then curl -sf --max-time 5 "http://${HOST_IP}:8000/docs" >/dev/null \ && docker ps --format '{{.Names}}' \ | grep -qE '^(vss-rtvi-cv|vss-rtvi-vlm)$' fi ``` If the Alert Bridge probe fails, ask which mode to deploy and hand off to `the `/vss-build-vision-ai` stock Alerts workflow in the matching mode` (decline → stop; pre-authorized autonomous deploy → run directly with `verification` by default). If it passes, detect the mode per Step 1. --- ## The Two Modes (Deploy-Time Choice) | Mode | Deploy flag | Env (`.env`) | What runs | What is available | |---|---|---|---|---| | **CV (verification)** | `-m verification` | `MODE=2d_cv` | RT-CV (Grounding DINO) + Behavior Analytics + `alert-bridge` VLM verifier + **`rtvi-vlm`** | Static CV pipeline (**Workflow A**) + verification results & verdicts (**Workflow B**) + on-demand verification (**Workflow F**). VIOS webhooks register streams with RT-CV. Realtime rule CRUD (**D**) and Slack (**E**) are gated to real-time mode (skill refuses on CV). | | **VLM (real-time)** | `-m real-time` | `MODE=2d_vlm` | `alert-bridge` + `rtvi-vlm` | Dynamic VLM real-time alerts (**Workflow D**), Slack (**E**), incident queries (**C**), and always-on operation (**Workflow G** — `ALERT_AGENT_ALWAYS_ON=true` / `alert_agent.always_on: true` when deployed with `-m real-time`). No static CV pipeline. | **Switching modes** uses the `vss-build-vision-ai` teardown + deploy flow, asking for the other mode. Both modes use the same agent `config.yml`; VIOS webhooks follow `notification_config_${MODE}.json`. `rtvi-vlm` runs in both modes. **`RTVI_VLM_KAFKA_ENABLED` is mode-specific.** `overrides.env` ships `RTVI_VLM_KAFKA_ENABLED=false` for verification (`2d_cv`), where nothing consumes RT-VLM's Kafka output and leaving it on makes RT-VLM publish duplicate incidents that Logstash indexes under `mdx-vlm-incidents-1970-01-01`. Real-time (`2d_vlm`) alerts depend on RT-VLM publishing to Kafka, so the line must be commented out in that mode — `dev-profile.sh` does this automatically for `-m real-time`. If a real-time deployment produces no alerts, check that this override is not still active in `generated.env`. --- ## Step 1 — Detect the Currently Deployed Mode Before running any alert workflow, check which mode is live. **Kubernetes** — do not use `docker ps`. Prefer an explicit mode hint, then ask: ```bash # Operator / deploy hint (preferred on Kubernetes). # ALERTS_MODE=real-time|verification or MODE=2d_vlm|2d_cv if [ -n "${ALERTS_MODE:-}" ]; then case "${ALERTS_MODE}" in real-time|realtime|vlm|2d_vlm) echo "mode=VLM" ;; verification|cv|2d_cv) echo "mode=CV" ;; esac elif [ "${MODE:-}" = "2d_vlm" ]; then echo "mode=VLM" elif [ "${MODE:-}" = "2d_cv" ]; then echo "mode=CV" elif [ "${DEPLOYMENT_KIND:-docker}" = "kubernetes" ]; then # Docs walkthrough is real-time; ask rather than guessing from private pods. echo "Ask the user: is this alerts deployment real-time (VLM) or verification (CV)?" fi ``` **Docker only** — use CV-only containers as the signal (`vss-rtvi-vlm` runs in both modes, so it is **not** a reliable mode signal alone): ```bash if [ "${DEPLOYMENT_KIND:-docker}" != "kubernetes" ]; then # CV verification mode (vss-behavior-analytics + vss-rtvi-cv are CV-only) if docker ps --format '{{.Names}}' | grep -qx vss-behavior-analytics; then echo "mode=CV" # VLM real-time mode has no CV pipeline; vss-rtvi-vlm runs in both modes. elif docker ps --format '{{.Names}}' | grep -qx vss-rtvi-vlm; then echo "mode=VLM" fi fi ``` If `vss-behavior-analytics` is present → **CV mode** (which also has `vss-rtvi-vlm`). If only `vss-rtvi-vlm` is present (and no CV pipeline) → **VLM mode**. If neither matches on Docker, the alerts profile is not deployed — direct the user to the `vss-build-vision-ai` skill. Alternative Docker signal (preferred when `docker ps` isn't accessible): check the deployed `generated.env`, falling back to `overrides.env` before a deployment has generated one: ```bash if [ "${DEPLOYMENT_KIND:-docker}" != "kubernetes" ]; then ENV_FILE=deploy/docker/developer-profiles/dev-profile-alerts/generated.env [ -f "$ENV_FILE" ] || ENV_FILE=deploy/docker/developer-profiles/dev-profile-alerts/overrides.env grep -E '^MODE=' "$ENV_FILE" # MODE=2d_cv → CV mode (full superset) # MODE=2d_vlm → VLM real-time mode (vss-rtvi-vlm only; no vss-rtvi-cv) fi ``` --- ## Step 2 — Route by Deployed Mode | Deployed mode | User asks about… | Action | |---|---|---| | **VLM real-time** | Slack webhook setup/status/test/stop | **Workflow E** — `references/alert-notify.md` | | **VLM real-time** | always-on status ("is always-on active?"), always-on incidents, "why aren't always-on alerts appearing?" | **Workflow G** — `references/always-on.md` (operate only; config authoring is out of scope) | | **CV verification** | always-on operation | Refuse — always-on rides the realtime rule engine; canonical refusal text below | | **VLM real-time** | rule CRUD, or start/stop a realtime alert on a sensor (with **or without** a detection condition — no condition → default prompt), or stop/delete a named alert (by `alert_type`/condition or rule ID) | **Workflow D** — `references/alert-subscriptions.md` (incl. two-step stop/confirm) | | **CV verification** | subscription/rule CRUD or Slack/notification setup | Refuse — see canonical refusal text below | | **CV or VLM** | incident lookup / *what happened* (recent alerts, time-range, casual "any alerts today?") | **Workflow C (Query)** — works on both; **always run the query, never answer from memory** | | **CV** | verification results / verdicts ("was it confirmed?", "show verification results"), *how does verification work*, verifier-prompt customization | **Workflow B (Verification)** — `references/verification.md`. **But** a verdict/result **follow-up to an on-demand verification just run** ("was it confirmed?", "what was the result?") → stay in **Workflow F**: poll `/realtime/incidents` by the `correlationId` (that result is incident-kind, not in `mdx-vlm-alerts-*`) | | **CV** | one-shot "verify **this** clip/image" with a media URL, or the literal "on-demand" | **Workflow F (On-demand)** — `references/on-demand-verification.md` | | **CV** | static CV alert onboarding | **Workflow A (CV)** — onboard RTSP via `vss-manage-video-io-storage`; VIOS webhook registers it with RT-CV | | **VLM** | verification results / verdict inspection, verifier-prompt config, or on-demand verification (CV-only capabilities) | *Explain-only* asks → answer from **Workflow B/F** background, no calls needed. *Execution* asks → VLM-mode refusal text below (redeploy hint `-m verification`) | | **VLM** | a CV / behavior-analytics / PPE-rule alert needing the static CV pipeline | **Redeployment required** — confirm first, then `the `/vss-build-vision-ai` stock Alerts workflow in verification mode` | | **any** | video summarization, highlight reels, reports, non-alert analytics | **Out of scope** — hand off to `vss-generate-video-report` / `vss-query-analytics` (Cross-Skill Links); do **not** answer it via incidents or rules, even when incidents are empty | **Always confirm before triggering a redeploy.** A mode switch stops all currently-running monitoring and restarts services. ### Intent precedence (first match wins) 1. **Workflow E (Slack)** — Slack-specific keywords (`slack`, `webhook` + `slack`, `bot token`, `slack channel`). `notify` alone is **not** sufficient. 2. **Workflow F (On-demand)** — a one-shot "verify / check / analyze **this**" pointing at a **specific media artifact** (video/image URL, clip, file), or the literal `on-demand`. Guard: *continuous monitoring of a sensor/stream* is **never** F — that's D ("watch camera X for PPE" → D; "verify this clip URL for PPE" → F). 3. **Workflow G (Always-on)** — the literal `always-on` (status, incidents, troubleshooting phrasings). Operate-not-author: status checks and queries only; never author or edit always-on rule config. A request to *create* an ordinary realtime rule is **not** G — that's D.