菜单
Learns richer spatial representations by training on both 2D and 3D data simultaneously. Combines 3D intra-modal self-distillation with 2D-3D cross-modal joint embedding, achieving 14.2% and 4.8% improvements over single-modality baselines in scene understanding and geometric consistency.
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/ADu2021/skillXiv --skill concerto-joint-learning命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
正在显示 SKILL.md
基于 SOC 职业分类
| name | concerto-joint-learning |
| title | Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2510.23607 |
| keywords | ["Self-Supervised Learning","Multimodal","Spatial Understanding","3D Vision"] |
| description | Learns richer spatial representations by training on both 2D and 3D data simultaneously. Combines 3D intra-modal self-distillation with 2D-3D cross-modal joint embedding, achieving 14.2% and 4.8% improvements over single-modality baselines in scene understanding and geometric consistency. |
Human spatial understanding emerges from multiple sensory modalities simultaneously. Concerto mirrors this by learning from paired 2D images and 3D point clouds, discovering representations richer than either modality alone.
The joint learning approach creates spatial features with superior geometric and semantic consistency, improving downstream scene understanding tasks.
Key insight: simultaneous 2D-3D training creates more coherent spatial concepts than training separately. Concerto uses:
Create dual-stream encoders for 2D and 3D data. Each stream learns modality-specific representations while staying aligned through contrastive loss:
class ConcertoEncoder(nn.Module):
def __init__(self, feature_dim=256):
super().__init__()
# 2D encoder (e.g., Vision Transformer)
self.image_encoder = ViT(
patch_size=16,
num_layers=12,
hidden_dim=768
)
# 3D encoder (e.g., PointNet++)
self.pointcloud_encoder = PointNet(
num_layers=4,
feature_dim=feature_dim
)
# Projection heads for alignment
self.image_proj = nn.Sequential(
nn.Linear(768, 512),
nn.ReLU(),
nn.Linear(512, feature_dim)
)
.pc_proj = nn.Sequential(
nn.Linear(feature_dim, ),
nn.ReLU(),
nn.Linear(, feature_dim)
)
():
image_feat = .image_encoder(images)
pc_feat = .pointcloud_encoder(point_clouds)
image_proj = .image_proj(image_feat)
pc_proj = .pc_proj(pc_feat)
image_proj, pc_proj
Implement cross-modal contrastive learning that aligns 2D and 3D representations. Instances from the same scene should have similar features:
class CrossModalContrastiveLoss(nn.Module):
def __init__(self, temperature=0.07):
super().__init__()
self.temperature = temperature
def forward(self, image_proj, pc_proj):
"""Contrastive loss between 2D and 3D projections."""
# Normalize features
image_proj = torch.nn.functional.normalize(image_proj, dim=-1)
pc_proj = torch.nn.functional.normalize(pc_proj, dim=-1)
# Compute similarity matrix
batch_size = image_proj.shape[0]
logits = torch.mm(image_proj, pc_proj.t()) / self.temperature
# Labels: diagonal elements are positives (same scene)
labels = torch.arange(batch_size, device=image_proj.device)
# Symmetric cross-entropy loss
loss_i2p = torch.nn.functional.cross_entropy(logits, labels)
loss_p2i = torch.nn.functional.cross_entropy(logits.t(), labels)
return (loss_i2p + loss_p2i) / 2
Implement 3D intra-modal self-distillation using a momentum encoder for point clouds. This captures 3D geometry better than cross-modal learning alone:
class PointCloud3DDistillation(nn.Module):
def __init__(self, encoder, momentum=0.999):
super().__init__()
self.encoder = encoder
self.momentum_encoder = copy.deepcopy(encoder)
self.momentum = momentum
# Freeze momentum encoder
for param in self.momentum_encoder.parameters():
param.requires_grad = False
def forward(self, point_clouds, aug_point_clouds):
"""Self-distillation within 3D modality."""
# Online encoder
feat = self.encoder(point_clouds)
# Momentum encoder (no gradient)
with torch.no_grad():
feat_momentum = self.momentum_encoder(aug_point_clouds)
# Knowledge distillation loss
loss = torch.nn.functional.mse_loss(feat, feat_momentum.detach())
# Update momentum encoder
self._update_momentum_encoder()
return loss
def _update_momentum_encoder(self):
"""Update momentum encoder with EMA."""
for param, momentum_param in zip(
self.encoder.parameters(),
self.momentum_encoder.parameters()
):
momentum_param.data = (
self.momentum * momentum_param.data +
(1 - self.momentum) * param.data
)
| Parameter | Recommendation |
|---|---|
| Feature dimension | 256 (balance capacity and efficiency) |
| Temperature | 0.07-0.1 (standard for contrastive learning) |
| Momentum coefficient | 0.999 (slow teacher updates) |
| Cross-modal weight | 1.0, 3D distillation weight |
| Batch size | 256-512 (contrastive learning needs diverse negatives) |
When to use:
When NOT to use:
Common pitfalls:
Reference: Concerto on arXiv