Train video QA models on 6K examples using pure RL instead of costly supervised fine-tuning, then adaptively sample video frames at inference by monitoring answer consensus across multiple reasoning trajectories.
Train video QA models on 6K examples using pure RL instead of costly supervised fine-tuning, then adaptively sample video frames at inference by monitoring answer consensus across multiple reasoning trajectories.
Video-RTS: Data-Efficient Video Reasoning with Adaptive Test-Time Computation
Most video reasoning approaches waste resources: they collect massive labeled datasets and ignore compute reuse at inference. Video-RTS flips this paradigm. By using reinforcement learning directly on small datasets and adapting frame sampling density based on output consistency, the method achieves performance comparable to models trained on 169K examples using only 6K examples.
The key insight is two-fold: (1) outcome-based rewards from RL eliminate the need for expensive supervised reasoning annotations, and (2) at inference, you can cheaply determine when you have seen enough frames by checking if multiple model samples agree on the answer.
Core Concept
Video-RTS operates in two phases. During training, it applies Group Relative Policy Optimization (GRPO) with simple reward signals: accuracy (did you get the right answer?) and format (is your reasoning properly structured?). No need to label full reasoning chains. During inference, it starts with sparse frame sampling (32 frames for an hour-long video) and gradually densifies until multiple reasoning trajectories converge on the same answer. This voting mechanism stops computation when confident, avoiding wasteful processing of redundant frames.
Architecture Overview
Base Model: Qwen-2.5-VL-7B vision-language model for video understanding
GRPO Training: Group Relative Policy Optimization with dual rewards (accuracy + formatting)
Sparse Frame Sampling: Initially extract 32 uniformly spaced frames from video
Trajectory Ensemble: Generate multiple reasoning paths with varied frame selections
Consensus Voting: Check if model outputs agree; densify frames only if uncertain
Adaptive Stopping: Halt when majority vote is reached or frame budget exhausted
Implementation
Step 1: Prepare Video QA Dataset with Outcome Labels
Collect video QA examples with only question, video, and ground truth answer labels. No reasoning annotations needed:
import torch
from transformers import AutoProcessor, AutoModelForVision2Seq
from datasets import load_dataset
# Load lightweight dataset (6K examples)
dataset = load_dataset("video_qa", split="train")
# Example structure: only question, video, and answer
example = {
: ,
: ,
:
}
model = AutoModelForVision2Seq.from_pretrained(
)
processor = AutoProcessor.from_pretrained()
"video"
"path/to/video.mp4"
"question"
"What sport is being played?"
"answer"
"basketball"
# Ground truth only
# Initialize Qwen-2.5-VL-7B
"Qwen/Qwen2.5-VL-7B-Instruct"
"Qwen/Qwen2.5-VL-7B-Instruct"
# Model outputs chains of thought + final answer automatically
# GRPO will reward correct final answers and well-formatted outputs
Step 2: GRPO Training - Pure RL Approach
Train using Group Relative Policy Optimization, which compares outputs within a group to determine which are better. This eliminates the need for reference models:
from torch.optim import AdamW
import torch.nn.functional as F
defgrpo_loss(model, batch_inputs, batch_answers):
"""
Compute GRPO loss: compare outputs within groups for relative ranking.
Rewards: accuracy (correct answer?) + format (proper structure?).
"""# Generate multiple responses per input (group)
num_samples = 4
all_outputs = []
for _ inrange(num_samples):
outputs = model.generate(
**batch_inputs,
max_length=256,
do_sample=True,
temperature=0.7
)
all_outputs.append(outputs)
# Compute rewards for each output
accuracy_rewards = []
format_rewards = []
for output in all_outputs:
decoded = processor.decode(output, skip_special_tokens=True)
# Reward 1: Does output contain correct answer?
has_correct = any(
ans.lower() in decoded.lower()
for ans in batch_answers
)
accuracy = 1.0if has_correct else0.0# Reward 2: Is output properly formatted (has answer at end)?
has_answer_section = any(
phrase in decoded.lower()
for phrase in ["answer:", "final answer", "the answer is"]
)
format_score = 1.0if has_answer_section else0.5
accuracy_rewards.append(accuracy)
format_rewards.append(format_score)
# Combine rewards
all_rewards = [
0.7 * acc + 0.3 * fmt
for acc, fmt inzip(accuracy_rewards, format_rewards)
]
# Group Relative Loss: compare within group
all_rewards = torch.tensor(all_rewards, device=model.device)
mean_reward = all_rewards.mean()
advantages = all_rewards - mean_reward
# Standard policy gradient: log_prob * advantage
log_probs = F.log_softmax(torch.stack(all_outputs, dim=0), dim=-1)
loss = -(log_probs * advantages.unsqueeze(-1)).mean()
return loss
# Training loop
optimizer = AdamW(model.parameters(), lr=1e-6)
for epoch inrange(1):
for batch in dataloader:
inputs = processor(batch["video"], batch["question"],
return_tensors="pt")
loss = grpo_loss(model, inputs, batch["answer"])
loss.backward()
optimizer.step()
optimizer.zero_grad()
Step 3: Sparse-to-Dense Frame Sampling at Inference
At inference, adaptively increase frame density only when model outputs disagree. Use majority voting to determine convergence:
defadaptive_frame_sampling(model, processor, video_path, question,
max_frames=128, convergence_threshold=0.8):
"""
Sparse-to-dense sampling: start with 32 frames, increase until
multiple reasoning trajectories agree on the answer.
"""import cv2
# Read video
cap = cv2.VideoCapture(video_path)
total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
current_frame_count = 32
trajectory_outputs = []
while current_frame_count <= max_frames:
# Sample current_frame_count frames uniformly
frame_indices = torch.linspace(
0, total_frames - 1, current_frame_count
).long()
frames = []
for idx in frame_indices:
cap.set(cv2.CAP_PROP_POS_FRAMES, idx.item())
ret, frame = cap.read()
if ret:
frames.append(frame)
# Generate reasoning trajectory with current frames
video_tensor = torch.stack([
torch.from_numpy(f).float() / 255.0for f in frames
])
inputs = processor(
video=video_tensor,
text=question,
return_tensors="pt"
)
with torch.no_grad():
output = model.generate(
**inputs,
max_length=256,
do_sample=False,
temperature=0.0
)
decoded = processor.decode(output[0], skip_special_tokens=True)
trajectory_outputs.append(decoded)
# Check consensus: do N trajectories agree?iflen(trajectory_outputs) >= 5:
# Extract final answers from all trajectories
answers = [extract_final_answer(traj)
for traj in trajectory_outputs[-5:]]
# Majority votefrom collections import Counter
vote_counts = Counter(answers)
max_votes = vote_counts.most_common(1)[0][1]
agreement_ratio = max_votes / len(answers)
if agreement_ratio >= convergence_threshold:
cap.release()
return trajectory_outputs[-1]
# Double frame count for next iteration
current_frame_count = min(current_frame_count * 2, max_frames)
cap.release()
return trajectory_outputs[-1]
defextract_final_answer(text):
"""Extract the final answer from reasoning trajectory."""
lines = text.split('\n')
for line inreversed(lines):
ifany(phrase in line.lower()
for phrase in ['answer:', 'final answer', 'the answer is']):
return line.split(':')[-1].strip()
return text.split()[-1]
Step 4: Multi-Trajectory Voting
Generate multiple trajectories per input and use consensus voting to decide when to stop:
defmulti_trajectory_voting(model, processor, video_path, question,
num_trajectories=5, max_frames=128):
"""
Generate multiple reasoning trajectories and vote on final answer.
Stops early if consensus is reached before max_frames.
"""
trajectories = []
answers = []
for traj_id inrange(num_trajectories):
# Slight variation in frame sampling per trajectory
output_text = adaptive_frame_sampling(
model, processor, video_path, question,
max_frames=max_frames
)
trajectories.append(output_text)
final_answer = extract_final_answer(output_text)
answers.append(final_answer)
# Majority vote on final answerfrom collections import Counter
answer_votes = Counter(answers)
best_answer = answer_votes.most_common(1)[0][0]
confidence = answer_votes[best_answer] / num_trajectories
return {
"answer": best_answer,
"confidence": confidence,
"trajectories": trajectories
}
Practical Guidance
Hyperparameter
Recommended Value
Notes
Training Dataset Size
6K examples
Shows saturation; diminishing returns beyond this
LoRA Rank (if used)
32-64
For efficient training on single GPU
GRPO Learning Rate
1e-6
Conservative to avoid destabilizing policy
KL Coefficient
0.04
Prevents divergence from base model
Initial Frame Count
32
Sparse sampling for hour-long videos
Max Frame Count
64-128
Based on task complexity (knowledge vs general)
Convergence Threshold
0.8
Stop when 4/5 trajectories agree
Number of Trajectories
5
Balance between voting confidence and latency
Accuracy Reward Weight
0.7
Prioritize correct answers over format
Format Reward Weight
0.3
Secondary emphasis on reasoning clarity
When to use Video-RTS:
Video QA tasks where you have limited labeled data (< 10K examples)
Applications where inference compute is flexible (can adapt based on uncertainty)
Scenarios prioritizing data efficiency over latency
Multi-turn reasoning where showing your work matters
When NOT to use Video-RTS:
Real-time systems requiring strict latency bounds
Tasks where output must match specific format (medical reports, precise counts)
Short videos where frame count is already minimal
Data-rich domains (100K+ labeled examples) where supervised learning dominates
Common pitfalls:
Setting KL coefficient too high, keeping model too close to base version
Convergence threshold too strict (0.95+), causing excessive frame sampling
Frame sampling strategy not matching video type (sports vs dialogue scenes)
Forgetting to normalize video input (0-1 range vs 0-255)
Not handling videos shorter than initial frame count (32 frames)
Reference
Li, Y., Liu, J., Su, Y., & Shen, J. (2025). Video-RTS: Rethinking Reinforcement Learning and Test-Time Scaling for Video Reasoning. arXiv:2507.06485. https://arxiv.org/abs/2507.06485
