菜单
Generate videos at 29 FPS with interactive motion control through teacher-student distillation of motion-conditioned video models, using sliding-window causal attention and attention sinks to maintain constant latency for indefinite-length generation.
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/ADu2021/skillXiv --skill motion-stream-real-time-interactive-video命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
正在显示 SKILL.md
基于 SOC 职业分类
| name | motion-stream-real-time-interactive-video |
| title | MotionStream: Real-Time Video Generation with Interactive Motion Controls |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2511.01266 |
| keywords | ["Video Generation","Real-Time Rendering","Motion Control","Distillation","Attention Mechanisms"] |
| description | Generate videos at 29 FPS with interactive motion control through teacher-student distillation of motion-conditioned video models, using sliding-window causal attention and attention sinks to maintain constant latency for indefinite-length generation. |
Traditional video generation models either lack motion control or sacrifice speed for quality. MotionStream achieves both through a two-stage approach: a bidirectional teacher model with motion guidance is distilled into a fast causal student. The student uses sliding-window attention to handle arbitrary lengths at constant latency, with attention sinks preventing quality degradation during extended generation.
The result is interactive video generation where users paint trajectories and see results in real-time.
Real-Time Motion-Controlled Video Generation:
1. Design Motion Conditioning System
Encode user trajectories into guidance signals for video generation.
class MotionConditioner(nn.Module):
def __init__(self, hidden_dim=512):
# Sinusoidal embeddings for trajectory points
self.position_encoding = PositionalEncoding(hidden_dim)
# Process trajectories: list of (x, y, frame) points
self.trajectory_encoder = nn.LSTM(2, hidden_dim, batch_first=)
.projection = nn.Linear(hidden_dim, hidden_dim)
():
encoded = .position_encoding(trajectory_points)
lstm_out, _ = .trajectory_encoder(encoded)
motion_guidance = .projection(lstm_out)
motion_guidance
():
text_features = .encode_text(text)
trajectory :
motion_guidance = .encode_trajectory(trajectory)
combined = text_features + motion_guidance
:
combined = text_features
combined
2. Implement Teacher-Student Distillation
Train student to replicate teacher guidance with single forward pass.
class TeacherStudentDistillation(nn.Module):
def __init__(self, teacher_model, student_model):
self.teacher = teacher_model
self.student = student_model
def forward_teacher(self, condition, num_frames=16):
# Teacher processes bidirectionally for high quality
# Can look ahead and backward in sequence
video = self.teacher(condition, bidirectional=True)
return video
def forward_student(self, condition, num_frames=16):
# Student processes causally (left-to-right only)
video = self.student(condition, bidirectional=False)
return video
def compute_distillation_loss(self, condition, num_frames=16):
# Generate videos from both models
teacher_video = self.forward_teacher(condition, num_frames)
student_video = self.forward_student(condition, num_frames)
# Distribution matching distillation
# Minimize difference in frame-by-frame distributions
teacher_feat = self.teacher.encode(teacher_video) # [batch, frames, dim]
student_feat = self.student.encode(student_video)
# MSE on features
feature_loss = F.mse_loss(student_feat, teacher_feat)
# Perceptual loss on frames
teacher_frames = teacher_video
student_frames = student_video
perceptual_loss = self.compute_perceptual_loss(teacher_frames, student_frames)
return feature_loss + perceptual_loss
def compute_perceptual_loss(self, teacher_frames, student_frames):
# Use pretrained VGG for perceptual distance
teacher_perc = self.vgg(teacher_frames)
student_perc = self.vgg(student_frames)
return F.mse_loss(teacher_perc, student_perc)
3. Implement Sliding-Window Causal Attention
Enable constant-latency inference for arbitrary-length generation.
class SlidingWindowCausalAttention(nn.Module):
def __init__(self, hidden_dim, window_size=8, num_heads=8):
self.window_size = window_size
self.hidden_dim = hidden_dim
self.query_projection = nn.Linear(hidden_dim, hidden_dim)
self.key_projection = nn.Linear(hidden_dim, hidden_dim)
self.value_projection = nn.Linear(hidden_dim, hidden_dim)
self.output_projection = nn.Linear(hidden_dim, hidden_dim)
def forward(self, x, cache=None):
# x: [batch, 1, hidden_dim] (single frame during generation)
batch_size, seq_len, hidden_dim = x.shape
Q = self.query_projection(x) # [batch, 1, hidden_dim]
K = self.key_projection(x)
V = self.value_projection(x)
if cache is not None:
# Retrieve previous window of keys/values
past_K, past_V = cache
# Concatenate with current
K = torch.cat([past_K, K], dim=1) # [batch, window_size + 1, hidden_dim]
V = torch.cat([past_V, V], dim=1)
# Keep only last window_size frames
if K.shape[1] > self.window_size:
K = K[:, -self.window_size:, :]
V = V[:, -self.window_size:, :]
else:
K = K[:, :self.window_size, :]
V = V[:, :self.window_size, :]
# Scaled dot-product attention (no masking needed, only past is available)
scores = torch.matmul(Q, K.transpose(-2, -1)) / np.sqrt(hidden_dim)
weights = F.softmax(scores, dim=-1)
output = torch.matmul(weights, V)
output = self.output_projection(output)
# Cache for next iteration
cache = (K, V)
return output, cache
4. Implement Attention Sinks for Temporal Coherence
Prevent quality degradation during extended generation by anchoring to initial frames.
class AttentionSinkAttention(nn.Module):
def __init__(self, hidden_dim, num_sink_tokens=4):
self.num_sinks = num_sink_tokens
self.hidden_dim = hidden_dim
# Learnable sink tokens initialized from first frame
self.sink_tokens = nn.Parameter(torch.randn(1, num_sink_tokens, hidden_dim))
def forward(self, x, initial_frame=None):
# x: current video latents
# initial_frame: first frame (used as anchor)
batch_size, seq_len, hidden_dim = x.shape
# Initialize sinks from initial frame if provided
if initial_frame is not None:
self.sink_tokens.data = initial_frame[:, :self.num_sinks, :]
# Attention mechanism that always includes sinks
# sinks act as "memory anchors" preventing drift
# Compute attention over (current sequence + sinks)
combined = torch.cat([self.sink_tokens.expand(batch_size, -1, -1), x], dim=1)
# Attention computation (details omitted for brevity)
# Key: attention always sees sinks, preventing divergence from initial frame
return combined
5. Train with Self-Rollout During Training
Simulate inference-time behavior (rolling KV cache) during training.
def train_streaming_video_model(student_model, teacher_model, num_steps=10000):
optimizer = torch.optim.Adam(student_model.parameters(), lr=1e-4)
for step in range(num_steps):
condition = sample_condition_batch()
num_frames = 16
# Training with self-rollout (simulate streaming inference)
cache = None
student_loss = 0
for frame_idx in range(num_frames):
# Student forward (causal, streaming)
frame_student, cache = student_model.generate_frame(
condition, frame_idx, cache
)
# Teacher forward (for supervision)
full_video_teacher = teacher_model(condition)
frame_teacher = full_video_teacher[:, frame_idx, :, :, :]
# Distillation loss on this frame
loss_frame = F.mse_loss(frame_student, frame_teacher)
student_loss += loss_frame
optimizer.zero_grad()
student_loss.backward()
optimizer.step()
if step % 100 == 0:
print(f"Step {step}: Loss {student_loss.item():.4f}")
When to Use:
Hyperparameters:
When NOT to Use:
Pitfalls:
Integration Point: Deploy as interactive layer in video editing/creation tools.