Skip to main content الرئيسية المنشئون adu2021 skillxiv latent-particle-world-models
latent-particle-world-models Learn world models by decomposing scenes into latent particles with per-particle dynamics. Each particle represents an object with position, scale, and appearance. Learn distributed latent actions governing per-particle transitions, enabling multimodal video generation from identical initial conditions.
الانتقال إلى التثبيت سوق المهارات اكتشف واستكشف مهارات الذكاء الاصطناعي التي بناها المجتمع.
المهن ذات الصلة SOC
استنادا إلى تصنيف SOC المهني
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
عرض تفاصيل Prompt يتجاوز الأمر المباشر Prompt المخصّص للمراجعة. افحص المصدر قبل تشغيله.
npx skills add https://github.com/ADu2021/skillXiv --skill latent-particle-world-modelsيبقى الأمر في سطر واحد. مرّر أفقيًا لمراجعته كاملًا قبل النسخ.
تفضّل نسخة محلية؟ نزّل الملفات المتاحة حاليًا لدى SkillsMP.
تحميل Zip جاري التحميل... المزيد من هذا المستودع meaningful-kebab-case-name Convert arXiv papers into ready-to-use agent skills using category-aware extraction. First classifies the paper into one or more of 11 research categories, then applies a specialized extraction pipeline for each category — because different types of papers produce different types of usable knowledge. A single paper can yield multiple skills if it spans categories. Use this skill whenever the user wants to turn a paper into a skill, extract practical techniques from research, build a skill library from papers, convert arXiv papers into reusable agent instructions, or batch-process multiple papers into skills. Also trigger when someone asks about extracting actionable knowledge from papers, making research practical for LLM agents, or systematically converting academic contributions into structured agent capabilities.
action-quantization-behavior-cloning Establish regret bounds for behavior cloning with discretized actions combining statistical error and quantization error terms. Prove smoothness requirements for safe quantizer design, show that learning-based quantizers fail these requirements, and propose model-based augmentation to reduce error dependence from H² to H.
adaptive-lora-personalized-ranks Dynamically allocate LoRA ranks per-layer during fine-tuning instead of using fixed uniform ranks. Learn optimal rank for each layer and subject via variational framework with discretized exponential distribution, reducing memory footprint while maintaining fidelity and text-alignment.
name latent-particle-world-models title Latent Particle World Models: Self-Supervised Object-Centric Stochastic Dynamics Modeling version 0.0.2 engine skillxiv-v0.0.2-claude-opus-4.6 license MIT url https://arxiv.org/abs/2603.04553 keywords ["World Models","Object-Centric Learning","Latent Dynamics","Stochastic Generation","Video Understanding"] description Learn world models by decomposing scenes into latent particles with per-particle dynamics. Each particle represents an object with position, scale, and appearance. Learn distributed latent actions governing per-particle transitions, enabling multimodal video generation from identical initial conditions.
Latent Particle World Models: Object-Centric Stochastic Dynamics
Traditional world models predict full video frames (wasteful) or use global latent actions (insufficient for multi-entity scenes). Latent Particle World Models (LPWM) decomposes video into object-centric particles with independent dynamics. The key innovation is per-particle latent actions: instead of a single global action determining the full scene, each particle has its own latent action distribution, enabling rich multi-entity interactions and genuine stochastic behavior.
The core insight is that complex scenes with multiple agents naturally decompose into independent actors, each with their own latent policy. This enables realistic multi-modal generation where identical initial conditions lead to different outcomes (e.g., multiple agents choosing different actions).
Core Concept
LPWM operates through three coordinated mechanisms:
Object-Centric Decomposition : Segment video scenes into latent particles, each representing an object or agentPer-Particle Latent Actions : Learn distributions over latent actions specific to each particle, not global to the sceneFlexible Conditioning : Support diverse control modalities (language goals, external actions, user instructions) by mapping to per-particle latent actions
Architecture Overview
Input : Video sequences or action-annotated trajectoriesScene Decomposition : Latent VAE decomposes scenes into particles (position, scale, appearance)Dynamics Model : Temporal transitions with per-particle learned policiesLatent Action Sampler : Sample per-particle actions from learned distributionsReconstruction : Render predicted particle states back to image spaceOutput : Multimodal video predictions, controllable generation
Implementation Steps
Step 1: Design particle representation
Define compact per-particle state encoding.
class Particle :
"""
Represents a single object/entity in a scene.
"""
def __init__ (self, position, scale, appearance, opacity=1.0 ):
"""
position: (x, y) coordinates (normalized 0-1)
scale: (h, w) relative to image size
appearance: learned embedding (e.g., 16-dim vector)
opacity: alpha channel (0-1)
"""
self
self
self
self
def
to_state_vector
self
"""Concatenate into single state vector."""
return
self
self
self
self
@classmethod
def
from_state_vector
cls, state_vector
"""Reconstruct particle from state vector."""
2
2
4
4
20
20
return
class
SceneDecomposition
"""Decompose video frames into particles."""
def
__init__
self, num_particles=4 , video_vae=None
self
self
def
decompose_frame
self, frame
"""
Extract particles from a single frame.
"""
self
self
self
for
in
self
self
self
self
return
def
_slot_attention
self, latent, num_slots
"""Decompose latent into K slot representations."""
return
64
def
_decode_position
self, slot
"""Extract position coordinates from slot."""
return
0
1.0
1
1.0
def
_decode_scale
self, slot
"""Extract scale from slot."""
return
0.1
2
1.0
0.8
0.1
3
1.0
0.8
def
_decode_appearance
self, slot
"""Extract appearance embedding from slot."""
return
4
20
def
_decode_opacity
self, slot
"""Extract opacity from slot."""
return
0.5
20
1.0
0.5
Step 2: Model per-particle latent action distributions
Define learned stochastic policies for each particle.
class ParticleLatentPolicy :
"""
Learn distribution over latent actions for a specific particle type.
"""
def __init__ (self, latent_action_dim=4 ):
self .latent_action_dim = latent_action_dim
self .policy_net = nn.Sequential(
nn.Linear(21 , 64 ),
nn.ReLU(),
nn.Linear(64 , 64 ),
nn.ReLU(),
nn.Linear(64 , latent_action_dim * 2 )
)
def sample_action (self, particle_state ):
"""
Sample latent action for this particle.
Returns: latent action vector (e.g., 4D)
"""
output = self .policy_net(particle_state)
mean = output[:self .latent_action_dim]
log_std = output[self .latent_action_dim:]
std = torch.exp(log_std)
epsilon = torch.randn_like(std)
latent_action = mean + std * epsilon
return latent_action
def compute_logprob (self, particle_state, latent_action ):
"""Compute log-probability of action under this policy."""
output = self .policy_net(particle_state)
mean = output[:self .latent_action_dim]
log_std = output[self .latent_action_dim:]
std = torch.exp(log_std)
logprob = -0.5 * ((latent_action - mean) / std) ** 2 - log_std
return logprob.sum ()
class DynamicsModel :
"""
Update particle states based on latent actions.
z_i^{t+1} = f(z_i^t, a_i^t, context)
"""
def __init__ (self, state_dim=21 , latent_action_dim=4 ):
self .state_dim = state_dim
self .latent_action_dim = latent_action_dim
self .transition_net = nn.Sequential(
nn.Linear(state_dim + latent_action_dim, 128 ),
nn.ReLU(),
nn.Linear(128 , 128 ),
nn.ReLU(),
nn.Linear(128 , state_dim)
)
def predict_next_state (self, particle_state, latent_action ):
"""Predict next particle state given current state and action."""
input_vector = torch.cat([particle_state, latent_action], dim=-1 )
state_delta = self .transition_net(input_vector)
next_state = particle_state + state_delta
next_state = self ._apply_constraints(next_state)
return next_state
def _apply_constraints (self, state ):
"""Enforce valid particle state ranges."""
state[:, :2 ] = torch.clamp(state[:, :2 ], 0 , 1 )
state[:, 2 :4 ] = torch.clamp(state[:, 2 :4 ], 0.05 , 0.95 )
state[:, -1 ] = torch.clamp(state[:, -1 ], 0 , 1 )
return state
Step 3: Implement scene rendering from particles
Convert particle states back to image space.
class ParticleRenderer :
"""Render particles back to image space."""
def __init__ (self, image_size=64 ):
self .image_size = image_size
def render_scene (self, particles ):
"""
Compose particles into single image.
"""
canvas = np.zeros((self .image_size, self .image_size, 3 ))
sorted_particles = sorted (particles,
key=lambda p: np.mean(p.appearance))
for particle in sorted_particles:
canvas = self ._composite_particle(canvas, particle)
return canvas
def _composite_particle (self, canvas, particle ):
"""Composite single particle onto canvas with alpha blending."""
texture = self ._appearance_to_texture(particle.appearance)
center_x = int (particle.position[0 ] * self .image_size)
center_y = int (particle.position[1 ] * self .image_size)
h = int (particle.scale[0 ] * self .image_size)
w = int (particle.scale[1 ] * self .image_size)
x_min = max (0 , center_x - w // 2 )
x_max = min (self .image_size, center_x + w // 2 )
y_min = max (0 , center_y - h // 2 )
y_max = min (self .image_size, center_y + h // 2 )
if x_max > x_min and y_max > y_min:
texture_region = texture[:h, :w]
alpha = particle.opacity
canvas[y_min:y_max, x_min:x_max] = \
(1 - alpha) * canvas[y_min:y_max, x_min:x_max] + \
alpha * texture_region[:y_max - y_min, :x_max - x_min]
return canvas
def _appearance_to_texture (self, appearance_embedding ):
"""Convert appearance embedding to rendered texture."""
np.random.seed(hash (appearance_embedding.tobytes()) % (2 ** 32 ))
return np.random.randn(32 , 32 , 3 ) * 0.5 + 0.5
Step 4: Define training objective
Train to reconstruct videos and match target distributions.
def compute_lpwm_loss (model, video_batch, latent_action_dim=4 ):
"""
LPWM training loss combining reconstruction and KL divergence.
"""
batch_size, seq_len, H, W, C = video_batch.shape
first_particles = model.decomposer.decompose_frame(video_batch[:, 0 ])
total_loss = 0.0
for t in range (seq_len - 1 ):
current_frame = video_batch[:, t]
target_frame = video_batch[:, t + 1 ]
particles = model.decomposer.decompose_frame(current_frame)
latent_actions = []
log_probs = []
for i, particle in enumerate (particles):
particle_state = torch.tensor(particle.to_state_vector(), dtype=torch.float32)
action = model.policies[i].sample_action(particle_state)
latent_actions.append(action)
logprob = model.policies[i].compute_logprob(particle_state, action)
log_probs.append(logprob)
next_particles = []
for i, (particle, action) in enumerate (zip (particles, latent_actions)):
particle_state = torch.tensor(particle.to_state_vector(), dtype=torch.float32)
next_state = model.dynamics.predict_next_state(particle_state, action)
next_particle = Particle.from_state_vector(next_state.detach().numpy())
next_particles.append(next_particle)
predicted_frame = model.renderer.render_scene(next_particles)
recon_loss = torch.nn.functional.mse_loss(
torch.tensor(predicted_frame),
torch.tensor(target_frame)
)
kl_loss = -torch.mean(torch.stack(log_probs))
total_loss += recon_loss + 0.01 * kl_loss
return total_loss / seq_len
Step 5: Generation and evaluation
Generate multimodal videos and benchmark on control tasks.
def generate_multimodal_video (model, initial_frame, num_steps=10 , num_samples=4 ):
"""
Generate multiple video trajectories from identical initial frame.
Demonstrates stochasticity from per-particle latent actions.
"""
initial_particles = model.decomposer.decompose_frame(initial_frame)
trajectories = []
for sample_idx in range (num_samples):
particles = initial_particles
frames = [initial_frame]
for step in range (num_steps):
latent_actions = [
model.policies[i].sample_action(
torch.tensor(p.to_state_vector(), dtype=torch.float32)
)
for i, p in enumerate (particles)
]
next_particles = [
Particle.from_state_vector(
model.dynamics.predict_next_state(
torch.tensor(p.to_state_vector(), dtype=torch.float32),
action
).detach().numpy()
)
for p, action in zip (particles, latent_actions)
]
next_frame = model.renderer.render_scene(next_particles)
frames.append(next_frame)
particles = next_particles
trajectories.append(frames)
return trajectories
Practical Guidance Hyperparameter Selection:
Number of particles : 2-6. More = finer detail; higher computational cost.Latent action dimension : 2-8. Higher = more expressive per-particle policies.Appearance embedding dimension : 8-16. Larger = more visual diversity.KL weight : 0.001-0.1. Higher = stronger stochasticity pressure.
Multi-agent or multi-entity scene modeling
Stochastic video generation where different outcomes are valid
Controllable generation with per-entity control
Settings requiring object-centric representations
Single-object scenes (global latent actions sufficient)
Deterministic environments where stochasticity is harmful
Real-time systems requiring fast generation
Scenes with many small, indistinct objects (decomposition becomes ambiguous)
Slot collapse : Some particle slots become inactive. Add auxiliary loss encouraging slot diversity.Poor appearance learning : If appearance embeddings are uninformative, increase embedding dimension or add contrastive loss.Policy convergence : Latent action policies may become deterministic (log_std → -∞). Add entropy regularization.Rendering artifacts : Alpha compositing can cause color oversaturation. Use pre-multiplied alpha or other blending modes.
Reference 