video-text-temporal-grounding

# Skill: video-text temporal grounding ## 1. Capability Definition & Real Case * **Professional Definition**: The ability to accurately localize and retrieve specific temporal intervals (start and end timestamps) within a continuous video stream that correspond to a natural language description. This requires the agent to align sequential visual frames with semantic concepts, maintain temporal coherence over sampled frames, and differentiate between adjacent or similar actions within a complex visual environment. * **Dimension Hierarchy**: Specialized Knowledge Navigation -> Corpus-Specific Retrieval & Reasoning -> video-text temporal grounding ### Real Case **[Case 1]** * **Initial Environment**: A RAG environment containing a news broadcast video as the primary retrieved asset. The video is represented as a sequence of dynamic resolution frames stored in a multimodal database. * **Real Question**: Select the clips of the video that contain the reporter discussing the specific details of the rescue mission mission. * **Real Trajectory**: The agent samples 8 frames uniformly across the video to establish a coarse timeline. It identifies high-resolution visual landmarks of a rescue boat at the 45-second mark. Using the temporal rotary position embedding (M-RoPE), it inspects adjacent frames to confirm the exact moment the discussion starts and ends. * **Real Answer**: 00:45 - 01:22 * **Why this demonstrates the capability**: This case demonstrates temporal grounding because the answer is not a single fact, but a duration. The agent must successfully navigate the visual sequence to define boundaries that satisfy the semantic constraints of the query. --- **[Case 2]** * **Initial Environment**: A multimodal corpus of instructional cooking videos (simulated from YouCook2/Breakfast datasets). Each video contains multiple discrete steps in a complex recipe. * **Real Question**: At what point in the video does the chef start adding the milk to the batter and when is the pouring finished? * **Real Trajectory**: The agent retrieves the 'Pancake Preparation' video. It performs iterative temporal narrowing, starting with 3D-convolution based motion detection to find the 'pouring' action and then zooming in on frames to identify the white liquid entity vs. the bowl batter. * **Real Answer**: 03:12 - 03:28 * **Why this demonstrates the capability**: Grounding requires distinguishing between 'preparing' to pour and the 'actual' pouring. The agent's ability to provide a precise [Start, End] interval shows it can resolve actions into structured temporal facts. ## Pipeline Execution Instructions To synthesize data for this capability, you must strictly follow a 3-phase pipeline. **Do not hallucinate steps.** Read the corresponding reference file for each phase sequentially: 1. **Phase 1: Environment Exploration** Read the exploration guidelines to discover raw knowledge seeds: `references/EXPLORATION.md` 2. **Phase 2: Trajectory Selection** Once Phase 1 is complete, read the selection criteria to evaluate the trajectory: `references/SELECTION.md` 3. **Phase 3: Data Synthesis** Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data: `references/SYNTHESIS.md`