vss-observability

# VSS Observability Use this only for `sample-applications/video-search-and-summarization`. Check `docker/compose.metrics-manager.yaml`, `config/nginx/nginx.conf`, `ui/react/src/components/Search/TelemetryAccordion.tsx`, `pipeline-manager/src/tracing.ts`, and `setup.sh` before giving operational advice. ## Environment setup Run the bundled bootstrap and work from the resolved app root: ```bash SKILL_DIR=".github/skills/vss-observability" APP_ROOT="$(bash "$SKILL_DIR/scripts/vss-bootstrap.sh")" cd "$APP_ROOT" ``` ## What observability exists VSS has two independent paths: 1. Live system and dataprep metrics use Metrics Manager. Set `ENABLE_METRICS_MANAGER=true` in a search-enabled Compose deployment. `docker/compose.metrics-manager.yaml` starts `docker.io/intel/metrics-manager:2026.2.0-20260715-weekly`. DataPrep sends `dataprep_embeddings_per_second` to its simple-metrics REST API, and the UI consumes its SSE stream through nginx. 2. Pipeline Manager traces use the Node OpenTelemetry SDK. `OTLP_TRACE_URL` selects an external OTLP HTTP endpoint; when it is empty, spans use the console exporter in Pipeline Manager logs. The service name is `videoSummary`. VSS does not bundle Jaeger, Tempo, an OpenTelemetry Collector, or a trace UI. ## Enable and verify live metrics ```bash ENABLE_METRICS_MANAGER=true source setup.sh --search # or ENABLE_METRICS_MANAGER=true source setup.sh --summary --search curl -f http://localhost:12345/metrics-manager/health curl -N http://localhost:12345/metrics-manager/metrics/stream curl http://localhost:9273/metrics | head ``` The nginx routes are deliberately limited to `/metrics-manager/health` and `/metrics-manager/metrics/stream`. The browser uses `EventSource`, reconnects automatically, and marks data stale when events stop. Metrics Manager provides CPU, RAM, GPU, and NPU values when the corresponding host devices exist; DataPrep publishes embedding throughput asynchronously. Missing accelerator metrics are valid on hosts without those devices. ## Enable traces ```bash OTLP_TRACE_URL=http://<trace-backend>:4318/v1/traces \ source setup.sh --summary --search ``` No credentials or headers are configured by VSS, so use an endpoint reachable without them or extend the trace exporter explicitly. ## Follow a video and find bottlenecks - Save `videoId` from `POST /manager/videos` and `summaryPipelineId` (the `stateId`) from `POST /manager/summary`. - Filter the trace backend for service `videoSummary` and the matching time window. Current auto-instrumented spans do not add those IDs as custom span attributes. - Inspect EVAM start/status requests for chunking, outbound VLM completion requests for captioning, outbound LLM completion requests for final summarization, and DataPrep calls for embedding/indexing. - Cross-check `/manager/summary/<stateId>/raw`, `/manager/pipeline/evam`, and `/manager/pipeline/frames`. - Compare slow DataPrep calls with `dataprep_embeddings_per_second` in the SSE stream. Compare model latency with CPU/RAM/GPU/NPU saturation. Metrics delivery failures do not imply trace failures, and missing metrics do not stop ingestion. `search-ms` does not currently have application-level OTel trace wiring. For API payloads, environment variables, and troubleshooting detail, read [references/telemetry-setup.md](references/telemetry-setup.md).