Diagnose a running or failing video-search-and-summarization deployment. Probes Pipeline Manager health and feature/config endpoints to detect whether the backend is up and which mode is live, then runs structured cross-service triage grounded in setup.sh, Docker Compose files, health routes, and OVMS config. Use when users say "is vss up", "what mode is running", "check vss health", "debug vss", "VSS isn't working", "OVMS won't start", "no summary appears", "search returns nothing", containers are crash-looping, healthchecks fail, or ports are conflicting in the VSS sample app.
Diagnose a running or failing video-search-and-summarization deployment. Probes Pipeline Manager health and feature/config endpoints to detect whether the backend is up and which mode is live, then runs structured cross-service triage grounded in setup.sh, Docker Compose files, health routes, and OVMS config. Use when users say "is vss up", "what mode is running", "check vss health", "debug vss", "VSS isn't working", "OVMS won't start", "no summary appears", "search returns nothing", containers are crash-looping, healthchecks fail, or ports are conflicting in the VSS sample app.
VSS Troubleshoot
Use this skill to diagnose a broken Video Search & Summarization deployment without guessing. The app is started with source setup.sh --summary, --search, --summary --search/--dual, or --summary-and-search/--unified; stopped with source setup.sh --down; user data reset with source setup.sh --clean-data.
Environment setup (run first)
This skill drives the Video Search & Summarization app through its real source
files, so the VSS application must be present and you must run commands from its
app root. Do this before anything else, and it works whether or not the VSS
source is already in your workspace.
Run the bundled bootstrap. It first tries to find an existing VSS checkout -
walking up from the current directory and inspecting the enclosing git repo - and
reuses it without ever re-cloning. Only when no checkout is found does it do a
shallow, single-branch, sparse checkout of just
sample-applications/video-search-and-summarization from main. It prints the
resolved app root on stdout:
# SKILL_DIR is THIS skill's own directory (shown to you when the skill loads);# in-repo it is .github/skills/vss-troubleshoot. Works the same if the skill is installed standalone.
SKILL_DIR=
APP_ROOT=
".github/skills/vss-troubleshoot"
"$(bash "$SKILL_DIR/scripts/vss-bootstrap.sh")"
cd
"$APP_ROOT"
Every command below assumes the working directory is this APP_ROOT. To pull
from a fork/branch or reuse a specific checkout dir, override VSS_REPO_URL,
VSS_REPO_BRANCH, or VSS_CLONE_DIR before running it.
First collect status
Run the read-only collector:
"$SKILL_DIR/scripts/triage.sh"
It prints Docker Compose/container status, tails recent logs, curls key health endpoints, checks documented host ports, and reports GPU/NPU device visibility. This matters because most failures are dependency chains: pipeline-manager depends on storage/database/search/summary services, and UI symptoms often originate in OVMS, vLLM, EVAM, VDMS, MinIO, RabbitMQ, or Postgres.
If Docker Compose cannot resolve services, run from the app root and compare with setup's own generated config:
Before diving into the decision tree, confirm whether the backend is even up and
which mode is live. Set HOST=http://${HOST_IP:-localhost}:${APP_HOST_PORT:-12345}
and run each command yourself, then relay the result. If nothing is deployed,
hand off to the vss-deploy skill at .github/skills/vss-deploy/SKILL.md.
# 1. Is the Pipeline Manager reachable?
curl -sf --max-time 5 "$HOST/manager/health" && echo" ← Pipeline Manager healthy" \
|| echo"UNREACHABLE - backend down or wrong HOST_IP/APP_HOST_PORT"# 2. Which capabilities/mode are live, and the resolved config
curl -s "$HOST/manager/app/features" | jq . # search/summary flags
curl -s "$HOST/manager/app/config" | jq . # resolved system config# 3. Subsystem probes
curl -s "$HOST/metrics-manager/health"# optional live metrics service
curl -s "$HOST/manager/audio/models" | jq . # whisper models (summary modes)
curl -s "$HOST/manager/pipeline/evam" | jq . # EVAM pipeline status
app/features returns string flags, not booleans -
{"summary":"FEATURE_ON","search":"FEATURE_OFF"} - so test against the string
(e.g. jq -e '.search=="FEATURE_ON"'). Use it to decide which workflow applies:
vss-search-index needs search==FEATURE_ON; vss-summarize-video needs
summary==FEATURE_ON. A backend that 404s on /manager/health while the model
servers (ovms-service, vllm-cpu-service, embedding server) are still loading
is usually starting, not broken - wait and re-probe.
Decision tree
1. Containers are missing, stopped, unhealthy, or crash-looping
Why: Compose depends_on gates many services on health. For example, summary mode needs ovms-service or vllm-cpu-service, video-ingestion, rabbitmq-service, and audio-analyzer; search mode needs multimodal-dataprep, vector-retriever, and multimodal-embedding-serving healthy.
Actions:
Read the first failing dependency's logs from triage.sh; later services often fail only because they waited for it.
Verify required environment variables from setup.sh: MinIO, Postgres, RabbitMQ credentials; VLM_MODEL_NAME, ENABLED_WHISPER_MODELS, OD_MODEL_NAME for summary; MULTIMODAL_EMBEDDING_MODEL for search; TEXT_EMBEDDING_MODEL for unified mode.
If containers start but app state is corrupt, only then consider source setup.sh --clean-data (this deletes Docker volumes listed by setup, including MinIO/Postgres/VDMS/data-prep data).
Model-download REST 8640 is loopback-only and transient while setup.sh
downloads missing summary-path models; it should not remain running afterward.
Why: Compose publishes these host ports. If another process owns one, the container may fail to bind or the UI may talk to the wrong service.
Actions:
Use the port section in triage.sh to identify listeners.
Stop the conflicting process or override the corresponding environment variable before rerunning source setup.sh ....
Curl http://localhost:3001/health for pipeline-manager and http://localhost:7890/health for video-search when applicable.
3. OVMS will not start or final summary is stuck
Inspect ovms-service logs and ov_models/ovms/config.json. Converted models
live under
ov_models/ovms/openvino_models/<device>/<precision>/<source-model>; setup
registers storage-aware names such as
Qwen_Qwen2.5-VL-3B-Instruct_CPU_int8.
Why: pipeline-manager sends VLM/LLM requests to http://ovms-service/v3 when ENABLE_VLLM is false. If OVMS is unhealthy, summary jobs can remain Ready or In Progress.
Likely fixes:
Incomplete or incompatible host model cache: identify the affected model
entry and directory first, stop VSS, then remove only that model directory
and rerun setup so model-download recreates it. Do not delete all
ov_models/ content unless the user accepts re-downloading every model.
Token limit error like prompt tokens + max tokens exceed model length: lower PM_SUMMARIZATION_MAX_COMPLETION_TOKENS below the default 4000, or use a model with a larger context window.
CL_OUT_OF_RESOURCES or cache at 100%: split VLM/LLM across CPU/GPU, use smaller/quantized models, or tune OVMS_CACHE_SIZE_GB cautiously.
NPU errors: verify the model supports NPU; otherwise set VLM_TARGET_DEVICE=CPU or another supported device.
4. Setup fails during model download
The model-download container runs before Compose only when an OD artifact or an
OVMS VLM/split LLM artifact is missing. Inspect the error's
ov_models/model-download-*.log, or docker logs vss-model-download while the
job is still running. Verify MODEL_DOWNLOAD_IMAGE, proxy settings, optional
Hugging Face token, selected model/device/precision, free disk space, and that
MODEL_DOWNLOAD_HOST_PORT (default 8640) is available. Increase
MODEL_DOWNLOAD_JOB_TIMEOUT from its 5400 second default only when a valid
download/conversion legitimately needs longer.
5. vLLM backend fails
When ENABLE_VLLM=true, setup adds compose.vllm.yaml, starts
vllm-cpu-service on host port 8200, and points VLM/LLM APIs to
http://vllm-cpu-service:8000/v1. Experimental ENABLE_VLLM_GPU=true instead
adds compose.vllm.xpu.yaml, starts vllm-xpu-service on the same default host
port, and points both APIs to http://vllm-xpu-service:8000/v1.
Why: In vLLM mode OVMS is not the active inference backend. Debugging OVMS logs will not explain vLLM request failures.
Actions: check the active vLLM service's /health endpoint and logs for model
download/context/cache problems. Verify VLM_MODEL_NAME,
HUGGINGFACE_TOKEN, and VLLM_MAX_MODEL_LEN; for CPU also inspect
VLLM_CPU_KVCACHE_SPACE, and for XPU inspect device visibility and
VLLM_GPU_MEM.
6. DLStreamer/EVAM pipeline errors or ingestion stalls
Check video-ingestion health (http://localhost:8090/pipelines) and logs, then rabbitmq-service and minio-service health/logs.
Why: summary ingestion uses DLStreamer Pipeline Server/EVAM to process video, publishes over RabbitMQ MQTT port 1883, and stores media through MinIO. If any of those fail, no chunks reach downstream summarization.
Actions:
Confirm OD_MODEL_NAME is a generic YOLO id supported by the model-download
Ultralytics plugin and that the IR exists under
ov_models/object-detection/ultralytics/public/<model>/FP32/.
Check GPU/device visibility if EVAM_DEVICE or detection uses accelerators.
Verify RabbitMQ credentials match RABBITMQ_USER/RABBITMQ_PASSWORD and MinIO credentials match MINIO_ROOT_USER/MINIO_ROOT_PASSWORD.
7. No summary appears
Follow this order: pipeline-manager health → Postgres health → MinIO health → video-ingestion → RabbitMQ → audio-analyzer → inference backend (ovms-service or vllm-cpu-service).
Why: pipeline-manager persists job state in Postgres, uses MinIO for assets, EVAM/RabbitMQ for video events, audio-analyzer for transcripts, and OVMS/vLLM for captions/final summaries.
Actions:
Look for Ready/In Progress stuck states and correlate with OVMS/vLLM logs.
For hallucinated or poor final summaries, try a larger VLM_MODEL_NAME; smaller models may have insufficient capacity.
On OpenCV/OpenGL/Mesa errors in summary/video processing, install libgl1-mesa-dri libgl1-mesa-dev, remove ov_models/ if needed, redeploy, and retest.
8. Search returns nothing
Check video-search (http://localhost:7890/health), multimodal-dataprep (/v1/dataprep/health on port 6016), vector-retriever (/ready on port 6008), vdms-vector-db (55555, or the Milvus stack when VECTORDB_BACKEND=milvus), multimodal-embedding-serving (9777), MinIO, and whether videos were actually ingested.
Why: search requires embeddings generated by multimodal-dataprep, stored in the active vector database under VS_INDEX_NAME (video_frame_embeddings for search, video_summary_embeddings for unified). video-search does not query the vector DB itself — it delegates all similarity search to vector-retriever, which embeds the query via multimodal-embedding-serving and reads the vector DB, then video-search aggregates the returned frames into ranked videos.
Actions:
If MULTIMODAL_EMBEDDING_MODEL or TEXT_EMBEDDING_MODEL changed, old vectors may have incompatible dimensions. Re-ingest, or reset data with source setup.sh --clean-data and rerun the correct setup mode.
Confirm the vector-retriever container is healthy and matches the backend: vector-retriever-vdms for VDMS, vector-retriever-milvus for Milvus (selected by VECTORDB_BACKEND). A mismatch, or a metric/index mismatch (VDB_METRIC_TYPE/VDB_INDEX_TYPE) between multimodal-dataprep and vector-retriever, yields empty results.
Check vector-retriever logs for the embedding call to multimodal-embedding-serving and the vector-DB read; check video-search logs for the delegation call to http://vector-retriever:8000/query.
Check accuracy settings: model dimensionality, FRAME_INTERVAL, ENABLE_OBJECT_DETECTION, and video diversity affect result quality.
Log and data locations
The Compose files do not define application log files; use Docker stdout/stderr
via docker logs or triage.sh. Important persistent locations are Docker
volumes docker_minio_data, docker_pg_data, docker_vdms-db,
docker_audio_analyzer_data, docker_data-prep, docker_collector_signals;
the host-backed OVMS repository at ov_models/ovms/; object detection models
under ov_models/object-detection/ultralytics/public/; and failed setup logs at
ov_models/model-download-*.log.
See references/common-failures.md for a compact symptom/cause/fix table.
