| name | video-reasoning-grounding |
| title | Open-o3 Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidence |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2510.20579 |
| keywords | ["video understanding","spatio-temporal grounding","explainability","reinforcement learning","visual reasoning"] |
| description | Ground video reasoning in explicit visual evidence by highlighting timestamps, objects, and bounding boxes, making reasoning verifiable and improving accuracy through RL rewards for spatio-temporal alignment. |
Technique: Spatio-Temporal Video Grounding — Evidence-Based Reasoning
Video understanding models often hallucinate or reason incorrectly because they lack explicit grounding in visual evidence. Open-o3-Video addresses this by requiring models to explicitly cite when (timestamps) and where (bounding boxes) key information appears, making reasoning traceable and verifiable.
Rather than accepting any plausible answer, the model is trained with RL rewards that encourage temporal precision (correct timestamps) and spatial precision (correct bounding boxes). This forces the model to truly understand video content instead of relying on spurious correlations.
Core Concept
Spatio-temporal grounding operates on three principles:
- Temporal Grounding: Model must cite specific timestamps for each reasoning step
- Spatial Grounding: Model must localize objects/regions using bounding boxes
- Verifiable Reasoning: Human or automated system can check if cited evidence actually supports the answer
- RL Rewards: Train with rewards for answer correctness AND evidence alignment
The result is more reliable reasoning that improves across various video understanding benchmarks through transparency and precision.
Architecture Overview
- Video Encoder: Extract frame features with temporal context
- Reasoning Module: Chain-of-thought generation that produces reasoning steps
- Temporal Localizer: Output timestamps for each reasoning step
- Spatial Localizer: Generate bounding boxes for relevant objects
- Evidence Verifier: Check if cited evidence actually supports reasoning
- RL Trainer: Optimize for answer + evidence alignment
Implementation Steps
The core innovation is augmenting generation to include explicit grounding. This example shows the reasoning-with-grounding pipeline.
import torch
import torch.nn as nn
from typing import List, Tuple, Dict
from dataclasses import dataclass
@dataclass
:
text:
confidence:
timestamp:
bounding_box: [, , , ]
object_label:
(nn.Module):
():
().__init__()
.num_frames = num_frames
.mlp = nn.Sequential(
nn.Linear(hidden_dim, ),
nn.ReLU(),
nn.Linear(, .num_frames)
)
.softmax = nn.Softmax(dim=-)
() -> [torch.Tensor, ]:
logits = .mlp(reasoning_hidden.unsqueeze())
probs = .softmax(logits[])
predicted_frame = torch.argmax(probs).item()
probs, predicted_frame
(nn.Module):
():
().__init__()
.bbox_predictor = nn.Sequential(
nn.Linear(hidden_dim, ),
nn.ReLU(),
nn.Linear(, num_objects * )
)
.object_classifier = nn.Sequential(
nn.Linear(hidden_dim, ),
nn.ReLU(),
nn.Linear(, num_objects)
)
() -> [torch.Tensor, torch.Tensor, ]:
bboxes = .bbox_predictor(reasoning_hidden.unsqueeze())[]
bboxes = bboxes.view(-, )
bboxes = torch.sigmoid(bboxes) * torch.tensor(frame.shape[:])
object_scores = .object_classifier(reasoning_hidden.unsqueeze())[]
object_scores = torch.softmax(object_scores, dim=)
top_object = torch.argmax(object_scores).item()
bboxes, object_scores, top_object
(nn.Module):
():
().__init__()
.video_encoder = video_encoder
.language_model = language_model
.temporal = temporal_localizer
.spatial = spatial_localizer
() -> [GroundedReasoningStep]:
torch.no_grad():
frame_features = [.video_encoder(frame) frame video_frames]
reasoning_steps = []
current_context = question
step (max_steps):
reasoning_text, reasoning_hidden = .language_model.generate_with_hidden(
current_context,
max_tokens=
)
frame_probs, frame_idx = .temporal(reasoning_hidden)
predicted_timestamp = (frame_idx / (video_frames)) * video_duration
frame = video_frames[frame_idx]
bboxes, obj_scores, obj_idx = .spatial(reasoning_hidden, frame)
top_bbox = bboxes[obj_idx].tolist()
step_obj = GroundedReasoningStep(
text=reasoning_text,
confidence=(torch.(frame_probs)),
timestamp=predicted_timestamp,
bounding_box=((x) x top_bbox),
object_label=
)
reasoning_steps.append(step_obj)
current_context += reasoning_text
reasoning_steps
() -> :
temporal_error = (predicted_step.timestamp - ground_truth_timestamp)
temporal_reward = (, - temporal_error / )
pred_box = predicted_step.bounding_box
gt_box = ground_truth_bbox
iou = compute_iou(pred_box, gt_box)
spatial_reward = iou
answer_reward = predicted_step.text_contains_answer
total_reward = (
* answer_reward +
* temporal_reward +
* spatial_reward
)
total_reward
():
optimizer = torch.optim.Adam(model.parameters(), lr=)
epoch (num_epochs):
epoch_reward =
example training_data:
video_frames = example[]
question = example[]
ground_truth_answer = example[]
ground_truth_grounding = example[]
reasoning_steps = model(video_frames, question)
total_reward =
step_idx, step (reasoning_steps):
step_idx ground_truth_grounding:
gt_ts, gt_bbox = ground_truth_grounding[step_idx]
reward = compute_grounding_reward(step, gt_ts, gt_bbox)
total_reward += reward
loss = -total_reward / (reasoning_steps)
optimizer.zero_grad()
loss.backward()
optimizer.step()
epoch_reward += total_reward
()
model
