菜单
Improve spatial reasoning in vision-language models through five automatically-formulated pretext tasks (patch shuffling, flipping, inpainting, depth ordering, 3D position) that require zero human annotation, using only RGB/RGB-D images and RL optimization for 4% accuracy gains.
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/ADu2021/skillXiv --skill spatial-ssrl-spatial-understanding-rl命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
基于 SOC 职业分类
正在显示 SKILL.md
| name | spatial-ssrl-spatial-understanding-rl |
| title | Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised RL |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2510.27606 |
| keywords | ["Spatial Reasoning","Self-Supervised Learning","Vision-Language Models","Reinforcement Learning","3D Understanding"] |
| description | Improve spatial reasoning in vision-language models through five automatically-formulated pretext tasks (patch shuffling, flipping, inpainting, depth ordering, 3D position) that require zero human annotation, using only RGB/RGB-D images and RL optimization for 4% accuracy gains. |
Vision-language models struggle with spatial relationships despite being trained on billions of image-text pairs. Spatial-SSRL fixes this by creating five pretext tasks that derive supervisory signals directly from image structure—no human annotation needed. Tasks range from simple (reorder shuffled patches) to geometric (predict 3D positions). The model learns through Group Relative Policy Optimization on automatically-generated question-answer pairs from existing datasets.
The key insight is that images contain all necessary geometry; you just need the right pretext tasks to make it explicit.
Self-Supervised Spatial Learning:
1. Define Five Pretext Tasks with Automatic Question Generation
Create task-specific question-answer generation from images.
class SpatialPretext:
@staticmethod
def shuffled_patch_reordering(image, patch_size=32, num_patches=16):
# Divide image into patches
patches = image.unfold(2, patch_size, patch_size).unfold(, patch_size, patch_size)
perm = torch.randperm(num_patches)
shuffled = patches[perm]
question =
answer =
question, answer
():
patches = image.unfold(, patch_size, patch_size).unfold(, patch_size, patch_size)
flip_idx = torch.randint((patches), (,)).item()
patches[flip_idx] = torch.flip(patches[flip_idx], dims=[-])
question =
answer =
question, answer
():
patches = image.unfold(, patch_size, patch_size).unfold(, patch_size, patch_size)
mask_idx :
mask_idx = torch.randint((patches), (,)).item()
masked_patches = patches.clone()
masked_patches[mask_idx] = torch.zeros_like(masked_patches[mask_idx])
candidates = [patches[mask_idx]] + [patches[torch.randint((patches), (,))] _ ()]
correct_idx =
question =
answer =
question, answer, candidates
():
h, w = image.shape[-:]
region_h, region_w = h // num_regions, w // num_regions
regions = []
i (num_regions):
region_depth = depth_map[i*region_h:(i+)*region_h, :].mean()
regions.append((i, region_depth.item()))
sorted_regions = (regions, key= x: x[])
question =
answer =
question, answer
():
pt1 = torch.tensor([image.shape[-]//, image.shape[-]//])
pt2 = torch.tensor([image.shape[-]*//, image.shape[-]*//])
depth1 = depth_map[pt1[], pt1[]]
depth2 = depth_map[pt2[], pt2[]]
question =
answer = depth1 < depth2
question, answer
2. Generate Spatial-SSRL-81k Dataset
Create automatic dataset from existing image collections.
def generate_spatial_dataset(image_dir, depth_dir, output_size=81000):
dataset = []
image_paths = list(Path(image_dir).glob("*.jpg"))[:output_size // 5]
for img_path in image_paths:
image = Image.open(img_path)
depth_path = Path(depth_dir) / (img_path.stem + ".npy")
depth = np.load(depth_path) if depth_path.exists() else None
# Generate 5 QA pairs per image
qa_pairs = [
SpatialPretext.shuffled_patch_reordering(image),
SpatialPretext.flipped_patch_recognition(image),
]
if depth is not None:
qa_pairs.extend([
SpatialPretext.regional_depth_ordering(image, depth),
SpatialPretext.relative_position_prediction(image, depth),
])
for q, a in qa_pairs:
dataset.append({
'image_path': str(img_path),
'question': q,
'answer': a
})
return dataset[:output_size]
3. Implement GRPO Training Loop
Train the model using Group Relative Policy Optimization.
class SpatialGRPOTrainer:
def __init__(self, model, tokenizer):
self.model = model
self.tokenizer = tokenizer
def compute_spatial_reward(self, generated_answer, reference_answer):
# Exact match for spatial reasoning
exact_match = float(generated_answer == reference_answer)
# Format compliance (answer in proper boxes)
has_box = '\\boxed{' in generated_answer
format_reward = float(has_box) if has_box else 0.1
return 0.9 * exact_match + 0.1 * format_reward
def train_step(self, batch, lr=1e-5):
images, questions, answers = batch
# Generate outputs
inputs = self.tokenizer(questions, images)
outputs = self.model.generate(inputs, max_length=256)
generated_answers = self.tokenizer.decode(outputs)
# Compute rewards per sample
rewards = [
self.compute_spatial_reward(gen, ref)
for gen, ref in zip(generated_answers, answers)
]
# Group Relative Policy Optimization
# Separate into groups by reward quartile
sorted_indices = np.argsort(rewards)
group_size = len(rewards) // 4
losses = []
for group_idx in range(4):
group_start = group_idx * group_size
group_end = (group_idx + 1) * group_size
group_indices = sorted_indices[group_start:group_end]
# Positive examples: high-reward samples
# Negative examples: low-reward samples
if group_idx >= 2:
# High reward group
loss = self.compute_grpo_loss(
inputs[group_indices], answers[group_indices], positive=True
)
else:
# Low reward group
loss = self.compute_grpo_loss(
inputs[group_indices], answers[group_indices], positive=False
)
losses.append(loss)
total_loss = sum(losses) / len(losses)
return total_loss.item()
def compute_grpo_loss(self, inputs, targets, positive=True):
outputs = self.model(**inputs)
logits = outputs.logits
# Standard language modeling loss
target_ids = self.tokenizer(targets).input_ids
loss = F.cross_entropy(logits, target_ids)
# Scale by relative position
return loss if not positive else loss * 0.5
4. Implement Cold-Start Supervised Fine-Tuning
Before RL, brief SFT stabilizes training.
def cold_start_sft(model, dataset, num_epochs=1):
optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5)
for epoch in range(num_epochs):
for batch in dataset:
images, questions, answers = batch
# Supervised next-token prediction
outputs = model(images=images, questions=questions, answers=answers)
loss = outputs.loss
optimizer.zero_grad()
loss.backward()
optimizer.step()
return model
When to Use:
Hyperparameters:
When NOT to Use:
Pitfalls:
Key Insight: Unlike typical self-supervised learning, you don't need paired data or external annotation. The geometric properties of images themselves provide infinite supervision.