vss-summarize-video

name	vss-summarize-video
description	Summarize a video through the VSS Pipeline Manager - start a summary pipeline with POST /summary (full required body), poll GET /summary/{stateId} until complete, then return the summary via GET /summary/{stateId}/raw. Use when the user says "summarize this video", "create a summary", "what happens in this video" (on an ingested video), or wants to run/inspect the summarization pipeline. Requires a summary-capable deployment (--summary, --dual, or --unified).
license	Apache-2.0
metadata	{"version":"1.0.0","tags":"vss operational summarization"}

VSS Summarize

Run the summarization pipeline via the Pipeline Manager. Run the curl commands yourself and relay results. Endpoints use the nginx /manager prefix.

Set HOST=http://${HOST_IP:-localhost}:${APP_HOST_PORT:-12345}.

Environment setup (run first)

This skill drives the Video Search & Summarization app through its real source files, so the VSS application must be present and you must run commands from its app root. Do this before anything else, and it works whether or not the VSS source is already in your workspace.

Run the bundled bootstrap. It first tries to find an existing VSS checkout - walking up from the current directory and inspecting the enclosing git repo - and reuses it without ever re-cloning. Only when no checkout is found does it do a shallow, single-branch, sparse checkout of just sample-applications/video-search-and-summarization from main. It prints the resolved app root on stdout:

# SKILL_DIR is THIS skill's own directory (shown to you when the skill loads);
# in-repo it is .github/skills/vss-summarize-video. Works the same if the skill is installed standalone.
SKILL_DIR=".github/skills/vss-summarize-video"
APP_ROOT="$(bash "$SKILL_DIR/scripts/vss-bootstrap.sh")"
cd "$APP_ROOT"

Every command below assumes the working directory is this APP_ROOT. To pull from a fork/branch or reuse a specific checkout dir, override VSS_REPO_URL, VSS_REPO_BRANCH, or VSS_CLONE_DIR before running it.

Preconditions

1. Backend healthy and summary enabled - probe first; if not, use vss-troubleshoot / vss-deploy:

   curl -sf "$HOST/manager/health" >/dev/null && \
   curl -s "$HOST/manager/app/features" | jq '.summary // .'   # confirm summary capability
   

2. A videoId to summarize - upload one with POST /manager/videos (multipart, field video), which returns { "videoId": "…" }. Or list existing - note the response is an object { "videos": [...] }, not a bare array, and name is a generated hash (the real filename is in url / dataStore.fileName):

   curl -s -X POST "$HOST/manager/videos" -F "video=@/path/to/clip.mp4" | jq .
   curl -s "$HOST/manager/videos" | jq '.videos[] | {videoId, file: .dataStore.fileName}'
   

1. Start the summary pipeline

POST /manager/summary. The body has required fields; missing any of title, sampling.*, or evam.evamPipeline returns 400. See references/summary-request.md for the full schema, prompt overrides, and audio options.

Minimal valid request:

curl -s -X POST "$HOST/manager/summary" \
  -H 'Content-Type: application/json' \
  -d '{
    "title": "Loading dock review",
    "videoId": "<VIDEO_ID>",
    "sampling": { "chunkDuration": 20, "samplingFrame": 5, "frameOverlap": 0, "multiFrame": 5 },
    "evam": { "evamPipeline": "object_detection" },
    "produceFinalSummary": true
  }' | jq .
# → { "summaryPipelineId": "<STATE_ID>" }

Sampling constraint: the Pipeline Manager enforces multiFrame == frameOverlap + samplingFrame. With frameOverlap: 0, set multiFrame == samplingFrame. Mismatch → 400 "Multi frame mismatch". evamPipeline is one of object_detection | video_ingestion.

2. Poll until complete

The returned summaryPipelineId is the stateId. GET /manager/summary/{stateId} has no top-level status/progress field (only /raw does) - progress lives in per-stage fields:

STATE_ID=<STATE_ID>
curl -s "$HOST/manager/summary/$STATE_ID" | jq '{
  chunking: .chunkingStatus,         # string, "complete" when chunked
  frames:   .frameSummaryStatus,     # COUNTS object: {complete, inProgress, na, ready}
  video:    .videoSummaryStatus,     # string: "na" → "inProgress" → "complete"  ← real done signal
  audio:    .audioTranscriptSummaryStatus,
  summary_len: (.summary | length)
}'

⚠️ Completion is videoSummaryStatus == "complete", NOT summary being non-empty. The final summary text is streamed in incrementally while videoSummaryStatus is still "inProgress", so polling on "summary length > 0" returns a truncated, mid-sentence result. Always gate on videoSummaryStatus. With produceFinalSummary: false there is no final stage - gate on frameSummaryStatus.inProgress == 0 instead.

until curl -s "$HOST/manager/summary/$STATE_ID" \
       | jq -e '.videoSummaryStatus == "complete"' >/dev/null; do sleep 10; done

Summarization is slow (VLM per-chunk + LLM map-reduce) - minutes, not seconds.

3. Retrieve the summary

curl -s "$HOST/manager/summary/$STATE_ID" | jq -r '.summary'   # final map-reduced summary
# Per-chunk captions live in .frameSummaries[] (each: frameKey, status, summary).
# NOT in .chunks[] - those only carry {chunkId, duration, audioTranscripts}:
curl -s "$HOST/manager/summary/$STATE_ID" | jq -r '.frameSummaries[] | "[\(.frameKey)] \(.summary)"'
curl -s "$HOST/manager/summary/$STATE_ID/raw" | jq .   # everything (audio, frames, status, …)

Present the final summary text; offer the per-chunk detail if useful. Audio with no speech yields an audioTranscriptSummary that says so - not an error.

Manage

curl -s "$HOST/manager/summary" | jq '.[] | {stateId, title}'   # list all
curl -s -X DELETE "$HOST/manager/summary/$STATE_ID"             # delete one