| name | video-text temporal grounding |
| description | Use this skill when a user wants to find a specific time window or 'clip' within a video based on an event, object, or action. Trigger it for requests like 'find the part in the video where they score', 'get the timestamps for the interview segment', 'show me the moment the cat jumps', or 'extract the clip of the engine failure'. This skill is for navigating continuous temporal visual data to locate discrete start and end markers (grounding). |
Skill: video-text temporal grounding
1. Capability Definition & Real Case
- Professional Definition: The ability to accurately localize and retrieve specific temporal intervals (start and end timestamps) within a continuous video stream that correspond to a natural language description. This requires the agent to align sequential visual frames with semantic concepts, maintain temporal coherence over sampled frames, and differentiate between adjacent or similar actions within a complex visual environment.
- Dimension Hierarchy: Specialized Knowledge Navigation -> Corpus-Specific Retrieval & Reasoning -> video-text temporal grounding
Real Case
[Case 1]
- Initial Environment: A RAG environment containing a news broadcast video as the primary retrieved asset. The video is represented as a sequence of dynamic resolution frames stored in a multimodal database.
- Real Question: Select the clips of the video that contain the reporter discussing the specific details of the rescue mission mission.
- Real Trajectory: The agent samples 8 frames uniformly across the video to establish a coarse timeline. It identifies high-resolution visual landmarks of a rescue boat at the 45-second mark. Using the temporal rotary position embedding (M-RoPE), it inspects adjacent frames to confirm the exact moment the discussion starts and ends.
- Real Answer: 00:45 - 01:22
- Why this demonstrates the capability: This case demonstrates temporal grounding because the answer is not a single fact, but a duration. The agent must successfully navigate the visual sequence to define boundaries that satisfy the semantic constraints of the query.
[Case 2]
- Initial Environment: A multimodal corpus of instructional cooking videos (simulated from YouCook2/Breakfast datasets). Each video contains multiple discrete steps in a complex recipe.
- Real Question: At what point in the video does the chef start adding the milk to the batter and when is the pouring finished?
- Real Trajectory: The agent retrieves the 'Pancake Preparation' video. It performs iterative temporal narrowing, starting with 3D-convolution based motion detection to find the 'pouring' action and then zooming in on frames to identify the white liquid entity vs. the bowl batter.
- Real Answer: 03:12 - 03:28
- Why this demonstrates the capability: Grounding requires distinguishing between 'preparing' to pour and the 'actual' pouring. The agent's ability to provide a precise [Start, End] interval shows it can resolve actions into structured temporal facts.
Pipeline Execution Instructions
To synthesize data for this capability, you must strictly follow a 3-phase pipeline. Do not hallucinate steps. Read the corresponding reference file for each phase sequentially:
-
Phase 1: Environment Exploration
Read the exploration guidelines to discover raw knowledge seeds:
references/EXPLORATION.md
-
Phase 2: Trajectory Selection
Once Phase 1 is complete, read the selection criteria to evaluate the trajectory:
references/SELECTION.md
-
Phase 3: Data Synthesis
Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data:
references/SYNTHESIS.md
