benchmark-video-summarization

name	benchmark-video-summarization
description	Benchmark a deployed LVS instance — set up test media, run single-file latency and burst-throughput tests, analyze GPU and latency metrics, and get configuration recommendations to improve performance.
license	Apache-2.0
metadata	{"version":"3.2.0","author":"NVIDIA Video Search and Summarization Team","github-url":"https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization","tags":"nvidia blueprint performance benchmarking lvs"}

Instructions

Follow the routing table and step-by-step workflow below. Execute each step in order on a first run. For repeat runs, go directly to the Repeat Runs section. Detailed reference material lives in references/ and benchmark scripts live in scripts/.

⚠️ Shared GPU systems: On a multi-user host, multiple LVS instances may be running on different ports. Always benchmark against the instance YOU deployed. Never assume port 38111 — always use the exact endpoint URL returned by vss-build-vision-ai when you deployed LVS. If you did not deploy an LVS instance in this session, deploy one first before running benchmarks.

Purpose

Measure the latency and throughput limits of a deployed LVS (Long Video Summarization) instance, identify GPU and pipeline bottlenecks, and suggest configuration changes that improve performance. Produces XLSX reports and JSON result files per scenario, plus a natural-language analysis with improvement recommendations.

Do NOT use this skill for:

• Deploying LVS — use the vss-build-vision-ai skill with the lvs profile.
• Production monitoring or alerting — this is an offline benchmarking tool.
• Non-LVS VSS profiles (RTVI, search-only, etc.) — the /files API and /summarize endpoint are LVS-specific.

Routing

Situation	Action
User has not deployed LVS in this session	Deploy LVS with vss-build-vision-ai (lvs profile) using a unique COMPOSE_PROJECT_NAME — note the exact endpoint URL it returns, then return here
LVS_BACKEND is not set	Ask the user for the endpoint URL of the LVS instance they deployed — do not guess or default
Results directory already exists with data and user asks to analyze only	Skip to Step 6: Analyze Results
First run or new hardware configuration	Run the full workflow Steps 0–7
Repeat run, setup already complete	Use run_benchmark.sh (see Repeat Runs section)

---

Prerequisites

Requirement	How to Check
LVS deployed by you via vss-build-vision-ai (lvs profile) with a unique COMPOSE_PROJECT_NAME	curl -sf ${LVS_BACKEND}/v1/ready returns 200
LVS_BACKEND set to your deployment's endpoint	echo $LVS_BACKEND (e.g. http://localhost:38111)
LVS_CONTAINER_NAME set to your LVS container name	echo $LVS_CONTAINER_NAME (e.g. vss-lvs)
VIA_DEV_API=true on YOUR LVS container	docker inspect ${LVS_CONTAINER_NAME} --format '{{range .Config.Env}}{{println .}}{{end}}' | grep VIA_DEV_API
NGC_API_KEY set in shell	echo $NGC_API_KEY (non-empty)
ngc CLI installed	ngc --version
Python 3.10+	python3 --version
ffprobe and ffmpeg installed	ffprobe -version && ffmpeg -version
Docker with Compose plugin	docker compose version

---

Deploy LVS for Benchmarking

Before benchmarking, deploy a fresh LVS instance using vss-build-vision-ai. On shared systems, always use a unique COMPOSE_PROJECT_NAME so your containers and volumes are isolated from other users.

# Choose a unique project name (e.g. your username)
export COMPOSE_PROJECT_NAME="lvs-bench-$(whoami)"

cd <repo>/deploy/docker

# REQUIRED FOR BENCHMARKING: the benchmark uploads videos via the LVS dev /files
# route, which is gated by VIA_DEV_API (default false; POST /files returns 404 when
# off). The lvs-server container loads its environment from
# services/video-summarization/.env (the compose `env_file:`), so VIA_DEV_API must
# be set THERE — putting it in any other env file will NOT reach the container.
# Set it before deploying:
grep -q '^VIA_DEV_API=' services/video-summarization/.env \
  && sed -i 's/^VIA_DEV_API=.*/VIA_DEV_API=true/' services/video-summarization/.env \
  || echo 'VIA_DEV_API=true' >> services/video-summarization/.env

# Deploy LVS on your designated GPUs
./scripts/dev-profile.sh up \
  --profile lvs \
  --hardware-profile RTXPRO6000BW \
  --llm-device-id <LLM_GPU> \
  --vlm-device-id <VLM_GPU>

# If LVS was already running, re-run the deploy command above so compose
# recreates the container with the new env (a plain `docker restart` does
# NOT re-read env_file changes).

Note the LVS endpoint (http://<HOST_IP>:38111) and container name: the compose file sets it statically to vss-lvs (verify with docker ps --filter name=lvs). Set these before continuing:

export LVS_BACKEND=http://localhost:38111
export LVS_CONTAINER_NAME=vss-lvs
export VLM_GPUS=<VLM_GPU>
export LLM_GPUS=<LLM_GPU>

Model downloads: First deployment downloads LLM (~20 GB) and VLM (~17 GB) model weights. This takes 20–40 minutes depending on network speed. Subsequent deployments reuse the volumes created under your COMPOSE_PROJECT_NAME and start in minutes.

---

Step 0: Pre-flight Check

On shared systems another tenant may hold port 38111 — so the benchmark must verify it is hitting YOUR instance and YOUR GPUs, not someone else's. preflight.sh does this (and fails fast if not).

Set your deployment's values, then run the pre-flight check:

export LVS_BACKEND=http://localhost:38111                          # YOUR LVS /summarize endpoint
export LVS_CONTAINER_NAME=vss-lvs                                   # YOUR LVS container (see deploy step)
export VLM_GPUS=<VLM_GPU>                                           # GPU(s) your VLM uses
export LLM_GPUS=<LLM_GPU>                                           # GPU(s) your LLM uses

./scripts/preflight.sh

preflight.sh exits non-zero unless all of the following hold:

• the config parses and vlm_gpus/llm_gpus are valid GPU ids within the host's GPU range;
• LVS_BACKEND is reachable (/v1/ready → 200) and the dev /files route is enabled (not 404);
• your LVS_CONTAINER_NAME actually owns the backend port — its server bound successfully (no address already in use in its logs) and no other container publishes that port;
• the configured VLM_GPUS/LLM_GPUS are reserved by your LVS's VLM/LLM containers — so you can't silently benchmark idle GPUs or another tenant's instance (the exact failure this guards against).

run_benchmark.sh runs preflight.sh automatically before every run; you can also run it standalone (above) any time.

If /files returns 404, the dev route is off — enable VIA_DEV_API=true on your LVS (see the deploy step) before continuing.

---

Step 1: Download Test Videos

The benchmark uses warehouse surveillance videos from the VSS sample dataset, hosted in NGC. scripts/fetch-videos.sh downloads the package and places a curated set — warehouse_4min.mp4, warehouse_5min.mp4, warehouse_10min.mp4 — into <VSS_BENCHMARK_DATA_DIR>/videos/, the exact filenames the default scripts/config.yaml references. It requires the ngc CLI and an NGC API key, and prints install/auth instructions if either is missing.

export VSS_BENCHMARK_DATA_DIR=${VSS_BENCHMARK_DATA_DIR:-$HOME/vss-benchmark-data}

# Idempotent; add FORCE=1 to re-fetch, or pass a package version (default 3.2.0)
./scripts/fetch-videos.sh

# Probe video durations to verify what was downloaded
find "${VSS_BENCHMARK_DATA_DIR}/videos" -name "*.mp4" | while read f; do
  dur=$(ffprobe -v quiet -show_entries format=duration -of csv=p=0 "$f" 2>/dev/null | cut -d. -f1)
  echo "$(basename $f): ${dur}s"
done

---

Step 2: Generate Test Clips (if needed)

Normally Step 1 provides real 5- and 10-minute clips and this step can be skipped. Use it only if a curated clip is missing (e.g. the NGC package layout changed) or you need custom durations: loop an available source video into the filenames the default scripts/config.yaml references — warehouse_5min.mp4 (300s) and warehouse_10min.mp4 (600s).

# Find a source video to loop from
SOURCE=$(find "${VSS_BENCHMARK_DATA_DIR}" -name "*.mp4" | head -1)
echo "Using source: ${SOURCE}"

# Create a videos/ subdirectory where the media server expects files
mkdir -p "${VSS_BENCHMARK_DATA_DIR}/videos"

for spec in warehouse_5min:300 warehouse_10min:600; do
  NAME="${spec%%:*}"; DURATION="${spec##*:}"
  OUT="${VSS_BENCHMARK_DATA_DIR}/videos/${NAME}.mp4"
  [ -f "$OUT" ] && echo "Already exists: ${OUT}" && continue
  ffmpeg -stream_loop -1 -i "${SOURCE}" -t ${DURATION} -c copy "${OUT}" -y -loglevel error
  echo "Created: ${OUT}"
done

# Verify clips
find "${VSS_BENCHMARK_DATA_DIR}/videos" -name "*.mp4" | while read f; do
  dur=$(ffprobe -v quiet -show_entries format=duration -of csv=p=0 "$f" 2>/dev/null | cut -d. -f1)
  echo "$(basename $f): ${dur}s"
done

---

Step 3: Start Media Server

The media server serves test videos over HTTP so the LVS /files endpoint can download them by URL.

SKILL_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

# Start the media server (nginx:alpine)
cd "${SKILL_DIR}"
docker compose -f scripts/media-server.yaml up -d
sleep 2

# Verify health
curl -sf "http://localhost:8888/health" && echo "Media server ready" || echo "Media server not ready"

# List available videos (JSON directory listing)
curl -s "http://localhost:8888/videos/" | python3 -m json.tool 2>/dev/null || \
  find "${VSS_BENCHMARK_DATA_DIR}/videos" -name "*.mp4" -exec basename {} \; | sed 's|^|http://localhost:8888/videos/|'

Note the filenames returned — you will need them in the next step to update config.yaml.

---

Step 4: Configure

Edit scripts/config.yaml with the actual video filenames discovered in Step 3:

1. Update video URLs — In both single_file_test and file_burst_test sections, replace HOST_IP with your host's LAN IP (hostname -I | awk '{print $1}' — NOT localhost; LVS downloads videos from inside its container) and make sure the filenames match those served by the media server (e.g., http://<HOST_IP>:8888/videos/warehouse_5min.mp4).

2. Set GPU assignments — Update vlm_gpus and llm_gpus to match your LVS deployment. Check YOUR container:

docker inspect "${LVS_CONTAINER_NAME}" \
  --format '{{range .Config.Env}}{{println .}}{{end}}' | grep -E "VLM_GPUS|LLM_GPUS|CUDA_VISIBLE"

3. Adjust chunk sizes — The default chunk_sizes: [10, 30] tests both 10-second and 30-second chunking. Larger chunks reduce API call overhead but increase per-chunk latency.

4. Adjust concurrency levels — The default concurrency_levels: [1, 2, 4, 8] for file_burst. Remove levels that exceed your hardware's memory capacity.

---

Step 5: Run Benchmark

SKILL_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
cd "${SKILL_DIR}/scripts"

# Create Python virtual environment if it doesn't exist
[ ! -d "vss-bench-env" ] && python3 -m venv vss-bench-env

# Activate venv and install requirements
source vss-bench-env/bin/activate
pip install -r requirements.txt -q

# LVS_BACKEND must already be set to YOUR deployment's endpoint
if [ -z "${LVS_BACKEND}" ]; then
  echo "ERROR: LVS_BACKEND is not set. Set it to your own LVS endpoint before running."
  exit 1
fi
export VIA_BACKEND="${LVS_BACKEND}"
export VIA_VLM_GPUS="${VLM_GPUS:?ERROR: VLM_GPUS must be set (e.g. export VLM_GPUS=6)}"
export VIA_LLM_GPUS="${LLM_GPUS:?ERROR: LLM_GPUS must be set (e.g. export LLM_GPUS=7)}"

# Run single_file scenario
python vss_perf_benchmark.py --config config.yaml --scenario single_file_test

# Run file_burst scenario (can be run separately or together)
python vss_perf_benchmark.py --config config.yaml --scenario file_burst_test

The benchmark creates an output directory (default: vss-perf-report/) with per-scenario subdirectories. Each scenario run generates an XLSX report and execution_summary.json.

Note: The output directory must not exist or must be empty before each run. Move or rename previous results before re-running:

mv vss-perf-report vss-perf-report-$(date +%Y%m%d-%H%M%S)

---

Step 6: Analyze Results

SKILL_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
OUTPUT_DIR="${SKILL_DIR}/vss-perf-report"

# Show top-level execution summary for single_file
echo "=== single_file_test execution summary ==="
cat "${OUTPUT_DIR}/single_file_test/execution_summary.json" | python3 -m json.tool

# Show per-test-case summaries
find "${OUTPUT_DIR}" -name "test_case_summary.json" | while read f; do
  echo ""
  echo "=== $(dirname $f | xargs basename) ==="
  cat "$f" | python3 -m json.tool
done

# List generated XLSX reports
find "${OUTPUT_DIR}" -name "*.xlsx" | while read f; do
  echo "Report: $f"
done

Read the JSON result files and extract the following key metrics for analysis:

• E2E latency (e2e_latency in seconds): total time from request to response
• VLM pipeline latency (vlm_pipeline_latency): time spent in the vision model pipeline
• VLM pipeline %: vlm_pipeline_latency / e2e_latency * 100
• CA-RAG latency (ca_rag_latency): context-aware RAG inference time
• VLM GPU utilization mean (vlm_gpu_usage_mean): GPU compute utilization % for VLM
• LLM GPU utilization mean (llm_gpu_usage_mean): GPU compute utilization % for LLM