Perception + memory + actions for video, live streams, and desktop sessions.
Use this skill when you need to:
When to Use
You need video or audio perception, indexing, search, or timeline editing from files, URLs, desktop sessions, or live streams.
The task involves timestamps, searchable evidence, subtitles, clips, overlays, or real-time monitoring alerts.
You want one workflow that combines ingestion, understanding, retrieval, and media actions.
1) Desktop Perception
Start/stop a desktop session capturing screen, mic, and system audio
Stream live context and store episodic session memory
Run real-time alerts/triggers on what's spoken and what's happening on screen
Produce session summaries, a searchable timeline, and playable evidence links
2) Video ingest + stream
Ingest a file or URL and return a playable web stream link
Transcode/normalize: codec, bitrate, fps, resolution, aspect ratio
3) Index + search (timestamps + evidence)
Build visual, spoken, and keyword indexes
Search and return exact moments with timestamps and playable evidence
Auto-create clips from search results
4) Timeline editing + generation
Subtitles: generate, translate, burn-in
Overlays: text/image/branding, motion captions
Audio: background music, voiceover, dubbing
Programmatic composition and exports via timeline operations
5) Live streams (RTSP) + monitoring
Connect RTSP/live feeds
Run real-time visual and spoken understanding and emit events/alerts for monitoring workflows
Common inputs
Local file path, public URL, or RTSP URL
Desktop capture request:
start / stop / summarize session
Desired operations: get context for understanding, transcode spec, index spec, search query, clip ranges, timeline edits, alert rules
Common outputs
Stream URL
Search results with timestamps and evidence links
Generated assets: subtitles, audio, images, clips
Event/alert payloads for live streams
Desktop session summaries and memory entries
Canonical prompts (examples)
"Start desktop capture and alert when a password field appears."
"Record my session and produce an actionable summary when it ends."
"Ingest this file and return a playable stream link."
"Index this folder and find every scene with people, return timestamps."
"Generate subtitles, burn them in, and add light background music."
"Connect this RTSP URL and alert when a person enters the zone."
Running Python code
Before running any VideoDB code, change to the project directory and load environment variables:
from dotenv import load_dotenv
load_dotenv(".env")
import videodb
conn = videodb.connect()
This reads VIDEO_DB_API_KEY from:
Environment (if already exported)
Project's .env file in current directory
If the key is missing, videodb.connect() raises AuthenticationError automatically.
Do NOT write a script file when a short inline command works.
When writing inline Python (python -c "..."), always use properly formatted code — use semicolons to separate statements and keep it readable. For anything longer than ~3 statements, use a heredoc instead:
Do NOT read, write, or handle the API key yourself. Always let the user set it.
Quick Reference
Upload media
# URL
video = coll.upload(url="https://example.com/video.mp4")
# YouTube
video = coll.upload(url="https://www.youtube.com/watch?v=VIDEO_ID")
# Local file
video = coll.upload(file_path="/path/to/video.mp4")
Transcript + subtitle
# force=True skips the error if the video is already indexed
video.index_spoken_words(force=True)
text = video.get_transcript_text()
stream_url = video.add_subtitle()
Search inside videos
from videodb.exceptions import InvalidRequestError
video.index_spoken_words(force=True)
# search() raises InvalidRequestError when no results are found.# Always wrap in try/except and treat "No results found" as empty.try:
results = video.search("product demo")
shots = results.get_shots()
stream_url = results.compile()
except InvalidRequestError as e:
if"No results found"instr(e):
shots = []
else:
raise
Scene search
import re
from videodb import SearchType, IndexType, SceneExtractionType
from videodb.exceptions import InvalidRequestError
# index_scenes() has no force parameter — it raises an error if a scene# index already exists. Extract the existing index ID from the error.try:
scene_index_id = video.index_scenes(
extraction_type=SceneExtractionType.shot_based,
prompt="Describe the visual content in this scene.",
)
except Exception as e:
match = re.search(r"id\s+([a-f0-9]+)", str(e))
ifmatch:
scene_index_id = match.group(1)
else:
raise# Use score_threshold to filter low-relevance noise (recommended: 0.3+)try:
results = video.search(
query="person writing on a whiteboard",
search_type=SearchType.semantic,
index_type=IndexType.scene,
scene_index_id=scene_index_id,
score_threshold=0.3,
)
shots = results.get_shots()
stream_url = results.compile()
except InvalidRequestError as e:
if"No results found"instr(e):
shots = []
else:
raise
Timeline editing
Important: Always validate timestamps before building a timeline:
start must be >= 0 (negative values are silently accepted but produce broken output)
Use ws_listener.py to capture WebSocket events during recording sessions. Desktop capture supports macOS only.
Quick Start
Start listener: python scripts/ws_listener.py &
Get WebSocket ID: cat /tmp/videodb_ws_id
Run capture code (see reference/capture.md for full workflow)
Events written to: /tmp/videodb_events.jsonl
Query Events
import json
events = [json.loads(l) for l inopen("/tmp/videodb_events.jsonl")]
# Get all transcripts
transcripts = [e["data"]["text"] for e in events if e.get("channel") == "transcript"]
# Get visual descriptions from last 5 minutesimport time
cutoff = time.time() - 300
recent_visual = [e for e in events
if e.get("channel") == "visual_index"and e["unix_ts"] > cutoff]
Do not use ffmpeg, moviepy, or local encoding tools when VideoDB supports the operation. The following are all handled server-side by VideoDB — trimming, combining clips, overlaying audio or music, adding subtitles, text/image overlays, transcoding, resolution changes, aspect-ratio conversion, resizing for platform requirements, transcription, and media generation. Only fall back to local tools for operations listed under Limitations in reference/editor.md (transitions, speed changes, crop/zoom, colour grading, volume mixing).
When to use what
Problem
VideoDB solution
Platform rejects video aspect ratio or resolution
video.reframe() or conn.transcode() with VideoConfig
Need to resize video for Twitter/Instagram/TikTok
video.reframe(target="vertical") or target="square"
