vss-troubleshoot

name

vss-troubleshoot

description

Diagnose a running or failing video-search-and-summarization deployment. Probes Pipeline Manager health and feature/config endpoints to detect whether the backend is up and which mode is live, then runs structured cross-service triage grounded in setup.sh, Docker Compose files, health routes, and OVMS config. Use when users say "is vss up", "what mode is running", "check vss health", "debug vss", "VSS isn't working", "OVMS won't start", "no summary appears", "search returns nothing", containers are crash-looping, healthchecks fail, or ports are conflicting in the VSS sample app.

VSS Troubleshoot

Use this skill to diagnose a broken Video Search & Summarization deployment without guessing. The app is started with source setup.sh --summary, --search, --summary --search/--dual, or --summary-and-search/--unified; stopped with source setup.sh --down; user data reset with source setup.sh --clean-data.

Environment setup (run first)

This skill drives the Video Search & Summarization app through its real source files, so the VSS application must be present and you must run commands from its app root. Do this before anything else, and it works whether or not the VSS source is already in your workspace.

Run the bundled bootstrap. It first tries to find an existing VSS checkout - walking up from the current directory and inspecting the enclosing git repo - and reuses it without ever re-cloning. Only when no checkout is found does it do a shallow, single-branch, sparse checkout of just sample-applications/video-search-and-summarization from main. It prints the resolved app root on stdout:

# SKILL_DIR is THIS skill's own directory (shown to you when the skill loads);
# in-repo it is .github/skills/vss-troubleshoot. Works the same if the skill is installed standalone.
SKILL_DIR=".github/skills/vss-troubleshoot"
APP_ROOT="$(bash "$SKILL_DIR/scripts/vss-bootstrap.sh")"
cd "$APP_ROOT"

Every command below assumes the working directory is this APP_ROOT. To pull from a fork/branch or reuse a specific checkout dir, override VSS_REPO_URL, VSS_REPO_BRANCH, or VSS_CLONE_DIR before running it.

First collect status

Run the read-only collector:

"$SKILL_DIR/scripts/triage.sh"

It prints Docker Compose/container status, tails recent logs, curls key health endpoints, checks documented host ports, and reports GPU/NPU device visibility. This matters because most failures are dependency chains: pipeline-manager depends on storage/database/search/summary services, and UI symptoms often originate in OVMS, vLLM, EVAM, VDMS, MinIO, RabbitMQ, or Postgres.

If Docker Compose cannot resolve services, run from the app root and compare with setup's own generated config:

source setup.sh --summary config     # or --search config / --summary-and-search config

Quick health & mode check

Before diving into the decision tree, confirm whether the backend is even up and which mode is live. Set HOST=http://${HOST_IP:-localhost}:${APP_HOST_PORT:-12345} and run each command yourself, then relay the result. If nothing is deployed, hand off to the vss-deploy skill at .github/skills/vss-deploy/SKILL.md.

# 1. Is the Pipeline Manager reachable?
curl -sf --max-time 5 "$HOST/manager/health" && echo "  ← Pipeline Manager healthy" \
  || echo "UNREACHABLE - backend down or wrong HOST_IP/APP_HOST_PORT"

# 2. Which capabilities/mode are live, and the resolved config
curl -s "$HOST/manager/app/features" | jq .   # search/summary flags
curl -s "$HOST/manager/app/config"   | jq .   # resolved system config

# 3. Subsystem probes
curl -s "$HOST/metrics-manager/health"          # optional live metrics service
curl -s "$HOST/manager/audio/models"   | jq .  # whisper models (summary modes)
curl -s "$HOST/manager/pipeline/evam"  | jq .  # EVAM pipeline status

app/features returns string flags, not booleans - {"summary":"FEATURE_ON","search":"FEATURE_OFF"} - so test against the string (e.g. jq -e '.search=="FEATURE_ON"'). Use it to decide which workflow applies: vss-search-index needs search==FEATURE_ON; vss-summarize-video needs summary==FEATURE_ON. A backend that 404s on /manager/health while the model servers (ovms-service, vllm-cpu-service, embedding server) are still loading is usually starting, not broken - wait and re-probe.

Decision tree

1. Containers are missing, stopped, unhealthy, or crash-looping

Check these exact services first: nginx, pipeline-manager, postgres-service, minio-service, ovms-service, vllm-cpu-service, vllm-xpu-service, video-ingestion, audio-analyzer, rabbitmq-service, video-search, vdms-vector-db, multimodal-dataprep, vector-retriever, multimodal-embedding-serving, and optional metrics-manager.

Why: Compose depends_on gates many services on health. For example, summary mode needs ovms-service or vllm-cpu-service, video-ingestion, rabbitmq-service, and audio-analyzer; search mode needs multimodal-dataprep, vector-retriever, and multimodal-embedding-serving healthy.

Actions:

• Read the first failing dependency's logs from triage.sh; later services often fail only because they waited for it.
• Verify required environment variables from setup.sh: MinIO, Postgres, RabbitMQ credentials; VLM_MODEL_NAME, ENABLED_WHISPER_MODELS, OD_MODEL_NAME for summary; MULTIMODAL_EMBEDDING_MODEL for search; TEXT_EMBEDDING_MODEL for unified mode.
• If containers start but app state is corrupt, only then consider source setup.sh --clean-data (this deletes Docker volumes listed by setup, including MinIO/Postgres/VDMS/data-prep data).

2. Port conflict or UI unreachable

Default host ports from setup.sh/Compose:

• UI/nginx 12345; pipeline-manager 3001; search-ms 7890
• OVMS REST/gRPC 8300/9300; vLLM 8200; EVAM 8090; audio 8999
• RabbitMQ AMQP/management/MQTT 5672/15672/1883
• MinIO API/console 4001/4002; Postgres 5432; VDMS 55555; multimodal-dataprep 6016; vector-retriever 6008; embedding service 9777; telemetry 9273
• Model-download REST 8640 is loopback-only and transient while setup.sh downloads missing summary-path models; it should not remain running afterward.

Why: Compose publishes these host ports. If another process owns one, the container may fail to bind or the UI may talk to the wrong service.

Actions:

• Use the port section in triage.sh to identify listeners.
• Stop the conflicting process or override the corresponding environment variable before rerunning source setup.sh ....
• Curl http://localhost:3001/health for pipeline-manager and http://localhost:7890/health for video-search when applicable.

3. OVMS will not start or final summary is stuck

Inspect ovms-service logs and ov_models/ovms/config.json. Converted models live under ov_models/ovms/openvino_models/<device>/<precision>/<source-model>; setup registers storage-aware names such as Qwen_Qwen2.5-VL-3B-Instruct_CPU_int8.

Why: pipeline-manager sends VLM/LLM requests to http://ovms-service/v3 when ENABLE_VLLM is false. If OVMS is unhealthy, summary jobs can remain Ready or In Progress.

Likely fixes:

• Incomplete or incompatible host model cache: identify the affected model entry and directory first, stop VSS, then remove only that model directory and rerun setup so model-download recreates it. Do not delete all ov_models/ content unless the user accepts re-downloading every model.
• Token limit error like prompt tokens + max tokens exceed model length: lower PM_SUMMARIZATION_MAX_COMPLETION_TOKENS below the default 4000, or use a model with a larger context window.
• CL_OUT_OF_RESOURCES or cache at 100%: split VLM/LLM across CPU/GPU, use smaller/quantized models, or tune OVMS_CACHE_SIZE_GB cautiously.
• NPU errors: verify the model supports NPU; otherwise set VLM_TARGET_DEVICE=CPU or another supported device.

4. Setup fails during model download

The model-download container runs before Compose only when an OD artifact or an OVMS VLM/split LLM artifact is missing. Inspect the error's ov_models/model-download-*.log, or docker logs vss-model-download while the job is still running. Verify MODEL_DOWNLOAD_IMAGE, proxy settings, optional Hugging Face token, selected model/device/precision, free disk space, and that MODEL_DOWNLOAD_HOST_PORT (default 8640) is available. Increase MODEL_DOWNLOAD_JOB_TIMEOUT from its 5400 second default only when a valid download/conversion legitimately needs longer.

5. vLLM backend fails

When ENABLE_VLLM=true, setup adds compose.vllm.yaml, starts vllm-cpu-service on host port 8200, and points VLM/LLM APIs to http://vllm-cpu-service:8000/v1. Experimental ENABLE_VLLM_GPU=true instead adds compose.vllm.xpu.yaml, starts vllm-xpu-service on the same default host port, and points both APIs to http://vllm-xpu-service:8000/v1.

Why: In vLLM mode OVMS is not the active inference backend. Debugging OVMS logs will not explain vLLM request failures.

Actions: check the active vLLM service's /health endpoint and logs for model download/context/cache problems. Verify VLM_MODEL_NAME, HUGGINGFACE_TOKEN, and VLLM_MAX_MODEL_LEN; for CPU also inspect VLLM_CPU_KVCACHE_SPACE, and for XPU inspect device visibility and VLLM_GPU_MEM.

6. DLStreamer/EVAM pipeline errors or ingestion stalls

Check video-ingestion health (http://localhost:8090/pipelines) and logs, then rabbitmq-service and minio-service health/logs.

Why: summary ingestion uses DLStreamer Pipeline Server/EVAM to process video, publishes over RabbitMQ MQTT port 1883, and stores media through MinIO. If any of those fail, no chunks reach downstream summarization.

Actions:

• Confirm OD_MODEL_NAME is a generic YOLO id supported by the model-download Ultralytics plugin and that the IR exists under ov_models/object-detection/ultralytics/public/<model>/FP32/.
• Check GPU/device visibility if EVAM_DEVICE or detection uses accelerators.
• Verify RabbitMQ credentials match RABBITMQ_USER/RABBITMQ_PASSWORD and MinIO credentials match MINIO_ROOT_USER/MINIO_ROOT_PASSWORD.

7. No summary appears

Follow this order: pipeline-manager health → Postgres health → MinIO health → video-ingestion → RabbitMQ → audio-analyzer → inference backend (ovms-service or vllm-cpu-service).

Why: pipeline-manager persists job state in Postgres, uses MinIO for assets, EVAM/RabbitMQ for video events, audio-analyzer for transcripts, and OVMS/vLLM for captions/final summaries.

Actions:

• Look for Ready/In Progress stuck states and correlate with OVMS/vLLM logs.
• For hallucinated or poor final summaries, try a larger VLM_MODEL_NAME; smaller models may have insufficient capacity.
• On OpenCV/OpenGL/Mesa errors in summary/video processing, install libgl1-mesa-dri libgl1-mesa-dev, remove ov_models/ if needed, redeploy, and retest.

8. Search returns nothing

Check video-search (http://localhost:7890/health), multimodal-dataprep (/v1/dataprep/health on port 6016), vector-retriever (/ready on port 6008), vdms-vector-db (55555, or the Milvus stack when VECTORDB_BACKEND=milvus), multimodal-embedding-serving (9777), MinIO, and whether videos were actually ingested.

Why: search requires embeddings generated by multimodal-dataprep, stored in the active vector database under VS_INDEX_NAME (video_frame_embeddings for search, video_summary_embeddings for unified). video-search does not query the vector DB itself — it delegates all similarity search to vector-retriever, which embeds the query via multimodal-embedding-serving and reads the vector DB, then video-search aggregates the returned frames into ranked videos.

Actions:

• If MULTIMODAL_EMBEDDING_MODEL or TEXT_EMBEDDING_MODEL changed, old vectors may have incompatible dimensions. Re-ingest, or reset data with source setup.sh --clean-data and rerun the correct setup mode.
• Confirm the vector-retriever container is healthy and matches the backend: vector-retriever-vdms for VDMS, vector-retriever-milvus for Milvus (selected by VECTORDB_BACKEND). A mismatch, or a metric/index mismatch (VDB_METRIC_TYPE/VDB_INDEX_TYPE) between multimodal-dataprep and vector-retriever, yields empty results.
• Check vector-retriever logs for the embedding call to multimodal-embedding-serving and the vector-DB read; check video-search logs for the delegation call to http://vector-retriever:8000/query.
• Check accuracy settings: model dimensionality, FRAME_INTERVAL, ENABLE_OBJECT_DETECTION, and video diversity affect result quality.

Log and data locations

The Compose files do not define application log files; use Docker stdout/stderr via docker logs or triage.sh. Important persistent locations are Docker volumes docker_minio_data, docker_pg_data, docker_vdms-db, docker_audio_analyzer_data, docker_data-prep, docker_collector_signals; the host-backed OVMS repository at ov_models/ovms/; object detection models under ov_models/object-detection/ultralytics/public/; and failed setup logs at ov_models/model-download-*.log.

See references/common-failures.md for a compact symptom/cause/fix table.