菜单
Train video QA models on 6K examples using pure RL instead of costly supervised fine-tuning, then adaptively sample video frames at inference by monitoring answer consensus across multiple reasoning trajectories.
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/ADu2021/skillXiv --skill video-rts-reinforcement-learning-test-time-scaling命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
正在显示 SKILL.md
| name | video-rts-reinforcement-learning-test-time-scaling |
| title | Video-RTS: Rethinking Reinforcement Learning and Test-Time Scaling for Video Reasoning |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2507.06485 |
| keywords | ["Video Reasoning","Reinforcement Learning","Test-Time Scaling","Data Efficiency","Sparse-to-Dense Sampling"] |
| description | Train video QA models on 6K examples using pure RL instead of costly supervised fine-tuning, then adaptively sample video frames at inference by monitoring answer consensus across multiple reasoning trajectories. |
Most video reasoning approaches waste resources: they collect massive labeled datasets and ignore compute reuse at inference. Video-RTS flips this paradigm. By using reinforcement learning directly on small datasets and adapting frame sampling density based on output consistency, the method achieves performance comparable to models trained on 169K examples using only 6K examples.
The key insight is two-fold: (1) outcome-based rewards from RL eliminate the need for expensive supervised reasoning annotations, and (2) at inference, you can cheaply determine when you have seen enough frames by checking if multiple model samples agree on the answer.
Video-RTS operates in two phases. During training, it applies Group Relative Policy Optimization (GRPO) with simple reward signals: accuracy (did you get the right answer?) and format (is your reasoning properly structured?). No need to label full reasoning chains. During inference, it starts with sparse frame sampling (32 frames for an hour-long video) and gradually densifies until multiple reasoning trajectories converge on the same answer. This voting mechanism stops computation when confident, avoiding wasteful processing of redundant frames.
Collect video QA examples with only question, video, and ground truth answer labels. No reasoning annotations needed:
import torch
from transformers import AutoProcessor, AutoModelForVision2Seq
from datasets import load_dataset
# Load lightweight dataset (6K examples)
dataset = load_dataset("video_qa", split="train")
# Example structure: only question, video, and answer
example = {
: ,
: ,
:
}
model = AutoModelForVision2Seq.from_pretrained(
)
processor = AutoProcessor.from_pretrained()
Train using Group Relative Policy Optimization, which compares outputs within a group to determine which are better. This eliminates the need for reference models:
from torch.optim import AdamW
import torch.nn.functional as F
def grpo_loss(model, batch_inputs, batch_answers):
"""
Compute GRPO loss: compare outputs within groups for relative ranking.
Rewards: accuracy (correct answer?) + format (proper structure?).
"""
# Generate multiple responses per input (group)
num_samples = 4
all_outputs = []
for _ in range(num_samples):
outputs = model.generate(
**batch_inputs,
max_length=256,
do_sample=True,
temperature=0.7
)
all_outputs.append(outputs)
# Compute rewards for each output
accuracy_rewards = []
format_rewards = []
for output in all_outputs:
decoded = processor.decode(output, skip_special_tokens=True)
# Reward 1: Does output contain correct answer?
has_correct = any(
ans.lower() in decoded.lower()
for ans in batch_answers
)
accuracy = 1.0 if has_correct else 0.0
# Reward 2: Is output properly formatted (has answer at end)?
has_answer_section = any(
phrase in decoded.lower()
for phrase in ["answer:", "final answer", "the answer is"]
)
format_score = 1.0 if has_answer_section else 0.5
accuracy_rewards.append(accuracy)
format_rewards.append(format_score)
# Combine rewards
all_rewards = [
0.7 * acc + 0.3 * fmt
for acc, fmt in zip(accuracy_rewards, format_rewards)
]
# Group Relative Loss: compare within group
all_rewards = torch.tensor(all_rewards, device=model.device)
mean_reward = all_rewards.mean()
advantages = all_rewards - mean_reward
# Standard policy gradient: log_prob * advantage
log_probs = F.log_softmax(torch.stack(all_outputs, dim=0), dim=-1)
loss = -(log_probs * advantages.unsqueeze(-1)).mean()
return loss
# Training loop
optimizer = AdamW(model.parameters(), lr=1e-6)
for epoch in range(1):
for batch in dataloader:
inputs = processor(batch["video"], batch["question"],
return_tensors="pt")
loss = grpo_loss(model, inputs, batch["answer"])
loss.backward()
optimizer.step()
optimizer.zero_grad()
At inference, adaptively increase frame density only when model outputs disagree. Use majority voting to determine convergence:
def adaptive_frame_sampling(model, processor, video_path, question,
max_frames=128, convergence_threshold=0.8):
"""
Sparse-to-dense sampling: start with 32 frames, increase until
multiple reasoning trajectories agree on the answer.
"""
import cv2
# Read video
cap = cv2.VideoCapture(video_path)
total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
current_frame_count = 32
trajectory_outputs = []
while current_frame_count <= max_frames:
# Sample current_frame_count frames uniformly
frame_indices = torch.linspace(
0, total_frames - 1, current_frame_count
).long()
frames = []
for idx in frame_indices:
cap.set(cv2.CAP_PROP_POS_FRAMES, idx.item())
ret, frame = cap.read()
if ret:
frames.append(frame)
# Generate reasoning trajectory with current frames
video_tensor = torch.stack([
torch.from_numpy(f).float() / 255.0 for f in frames
])
inputs = processor(
video=video_tensor,
text=question,
return_tensors="pt"
)
with torch.no_grad():
output = model.generate(
**inputs,
max_length=256,
do_sample=False,
temperature=0.0
)
decoded = processor.decode(output[0], skip_special_tokens=True)
trajectory_outputs.append(decoded)
# Check consensus: do N trajectories agree?
if len(trajectory_outputs) >= 5:
# Extract final answers from all trajectories
answers = [extract_final_answer(traj)
for traj in trajectory_outputs[-5:]]
# Majority vote
from collections import Counter
vote_counts = Counter(answers)
max_votes = vote_counts.most_common(1)[0][1]
agreement_ratio = max_votes / len(answers)
if agreement_ratio >= convergence_threshold:
cap.release()
return trajectory_outputs[-1]
# Double frame count for next iteration
current_frame_count = min(current_frame_count * 2, max_frames)
cap.release()
return trajectory_outputs[-1]
def extract_final_answer(text):
"""Extract the final answer from reasoning trajectory."""
lines = text.split('\n')
for line in reversed(lines):
if any(phrase in line.lower()
for phrase in ['answer:', 'final answer', 'the answer is']):
return line.split(':')[-1].strip()
return text.split()[-1]
Generate multiple trajectories per input and use consensus voting to decide when to stop:
def multi_trajectory_voting(model, processor, video_path, question,
num_trajectories=5, max_frames=128):
"""
Generate multiple reasoning trajectories and vote on final answer.
Stops early if consensus is reached before max_frames.
"""
trajectories = []
answers = []
for traj_id in range(num_trajectories):
# Slight variation in frame sampling per trajectory
output_text = adaptive_frame_sampling(
model, processor, video_path, question,
max_frames=max_frames
)
trajectories.append(output_text)
final_answer = extract_final_answer(output_text)
answers.append(final_answer)
# Majority vote on final answer
from collections import Counter
answer_votes = Counter(answers)
best_answer = answer_votes.most_common(1)[0][0]
confidence = answer_votes[best_answer] / num_trajectories
return {
"answer": best_answer,
"confidence": confidence,
"trajectories": trajectories
}
| Hyperparameter | Recommended Value | Notes |
|---|---|---|
| Training Dataset Size | 6K examples | Shows saturation; diminishing returns beyond this |
| LoRA Rank (if used) | 32-64 | For efficient training on single GPU |
| GRPO Learning Rate | 1e-6 | Conservative to avoid destabilizing policy |
| KL Coefficient | 0.04 | Prevents divergence from base model |
| Initial Frame Count | 32 | Sparse sampling for hour-long videos |
| Max Frame Count | 64-128 | Based on task complexity (knowledge vs general) |
| Convergence Threshold | 0.8 | Stop when 4/5 trajectories agree |
| Number of Trajectories | 5 | Balance between voting confidence and latency |
| Accuracy Reward Weight | 0.7 | Prioritize correct answers over format |
| Format Reward Weight | 0.3 | Secondary emphasis on reasoning clarity |
When to use Video-RTS:
When NOT to use Video-RTS:
Common pitfalls:
Li, Y., Liu, J., Su, Y., & Shen, J. (2025). Video-RTS: Rethinking Reinforcement Learning and Test-Time Scaling for Video Reasoning. arXiv:2507.06485. https://arxiv.org/abs/2507.06485
基于 SOC 职业分类
