菜单
Add explicit visual supervision to VLMs by training models to autoregressively reconstruct semantic image tokens, achieving 2-3% average gains and 10-point improvements on hallucination robustness.
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/ADu2021/skillXiv --skill semantic-visual-reconstruction命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
正在显示 SKILL.md
基于 SOC 职业分类
| name | semantic-visual-reconstruction |
| title | Autoregressive Semantic Visual Reconstruction Helps VLMs Understand Better |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2506.09040 |
| keywords | ["vision-language","semantic reconstruction","visual tokenization","multimodal supervision"] |
| description | Add explicit visual supervision to VLMs by training models to autoregressively reconstruct semantic image tokens, achieving 2-3% average gains and 10-point improvements on hallucination robustness. |
Traditional Vision-Language Models (VLMs) apply supervision only to text outputs while leaving rich visual input unsupervised. ASVR (Autoregressive Semantic Visual Reconstruction) addresses this asymmetry by training models to predict both semantic visual tokens and text tokens within a unified framework. This establishes a "perceptual foundation for image understanding" that improves robustness and reduces hallucinations.
Create tokenizer that captures high-level semantic features:
class SemanticVisualTokenizer:
def __init__(self, model_name: str = "vq-siglip"):
self.model_name = model_name
# Load pretrained semantic tokenizer
self.tokenizer = self._load_semantic_tokenizer(model_name)
self.vocab_size = self.tokenizer.config.vocab_size
def _load_semantic_tokenizer(self, model_name: str):
"""Load pretrained semantic visual tokenizer."""
# VQ-SigLIP: VQ-VAE variant using SigLIP embeddings
from transformers import AutoModel
return AutoModel.from_pretrained(
f"semantic-tokenizers/{model_name}",
trust_remote_code=
)
() -> torch.Tensor:
image = (image - image.mean()) / image.std()
torch.no_grad():
tokens = .tokenizer.encode(image)
tokens
() -> torch.Tensor:
torch.no_grad():
image = .tokenizer.decode(tokens)
image
Train model to predict both visual and text tokens:
class SemanticVisualVLM(torch.nn.Module):
def __init__(self, text_model, vision_encoder,
semantic_tokenizer: SemanticVisualTokenizer):
super().__init__()
self.text_model = text_model
self.vision_encoder = vision_encoder
self.semantic_tokenizer = semantic_tokenizer
# Output heads for both modalities
self.visual_head = torch.nn.Linear(
self.text_model.hidden_size,
semantic_tokenizer.vocab_size
)
self.text_head = torch.nn.Linear(
self.text_model.hidden_size,
self.text_model.vocab_size
)
def forward(self, image: torch.Tensor,
text_tokens: torch.Tensor,
visual_tokens: torch.Tensor) -> dict:
"""Forward pass with both visual and text supervision."""
# Encode image
image_features = self.vision_encoder(image)
# Interleave visual and text tokens for joint training
# [IMG_START] visual_tokens [IMG_END] text_tokens
combined_input = self._interleave_modalities(
image_features,
text_tokens
)
# Get hidden states
hidden_states = self.text_model(
combined_input,
output_hidden_states=True
).hidden_states
# Predict both visual and text tokens
visual_logits = self.visual_head(hidden_states)
text_logits = self.text_head(hidden_states)
return {
"visual_logits": visual_logits,
"text_logits": text_logits,
"hidden_states": hidden_states
}
def _interleave_modalities(self, image_features, text_tokens):
"""Create combined input with visual and text modality markers."""
# This is simplified; in practice, use proper sequence construction
return torch.cat([image_features, text_tokens], dim=0)
Stage 1: Pre-training on large-scale image-text data:
class PretrainingTrainer:
def __init__(self, model: SemanticVisualVLM):
self.model = model
self.optimizer = torch.optim.AdamW(
model.parameters(),
lr=1e-4
)
def compute_joint_loss(self, outputs: dict,
visual_tokens: torch.Tensor,
text_tokens: torch.Tensor) -> torch.Tensor:
"""Combined visual and text reconstruction loss."""
# Visual token prediction loss
visual_logits = outputs["visual_logits"]
visual_loss = torch.nn.functional.cross_entropy(
visual_logits.reshape(-1, self.model.semantic_tokenizer.vocab_size),
visual_tokens.reshape(-1)
)
# Text token prediction loss
text_logits = outputs["text_logits"]
text_loss = torch.nn.functional.cross_entropy(
text_logits.reshape(-1, self.model.text_model.vocab_size),
text_tokens.reshape(-1)
)
# Equal weighting for balance
total_loss = visual_loss + text_loss
return total_loss
def train_epoch(self, dataloader):
"""Train one epoch on large-scale data."""
total_loss = 0.0
for batch in dataloader:
images = batch["image"]
text_tokens = batch["text_tokens"]
# Encode images to semantic tokens
visual_tokens = self.model.semantic_tokenizer.encode_image(
images
)
# Forward pass
outputs = self.model(
images,
text_tokens,
visual_tokens
)
# Compute loss
loss = self.compute_joint_loss(
outputs,
visual_tokens,
text_tokens
)
# Update
self.optimizer.zero_grad()
loss.backward()
self.optimizer.step()
total_loss += loss.item()
return total_loss / len(dataloader)
Stage 2: Instruction tuning on diverse vision-language tasks:
class InstructionTuner:
def __init__(self, pretrained_model: SemanticVisualVLM):
self.model = pretrained_model
self.optimizer = torch.optim.AdamW(
self.model.parameters(),
lr=5e-5
)
def finetune_on_tasks(self, task_datasets: dict,
num_epochs: int = 3):
"""Fine-tune on diverse instruction-following tasks."""
for epoch in range(num_epochs):
total_loss = 0.0
total_samples = 0
for task_name, dataset in task_datasets.items():
for batch in dataset:
images = batch["image"]
instructions = batch["instruction"]
target_responses = batch["response"]
# Tokenize
text_tokens = self.model.text_model.tokenize(
instructions
)
response_tokens = self.model.text_model.tokenize(
target_responses
)
visual_tokens = (
self.model.semantic_tokenizer.encode_image(images)
)
# Forward
outputs = self.model(
images,
text_tokens,
visual_tokens
)
# Loss focuses on response prediction
response_logits = outputs["text_logits"][
len(text_tokens):
]
loss = torch.nn.functional.cross_entropy(
response_logits.reshape(
-1,
self.model.text_model.vocab_size
),
response_tokens.reshape(-1)
)
# Update
self.optimizer.zero_grad()
loss.backward()
self.optimizer.step()
total_loss += loss.item()
total_samples += 1
avg_loss = total_loss / total_samples
print(f"Epoch {epoch}: Loss = {avg_loss:.4f}")
Test improvements across multimodal understanding tasks:
def evaluate_model(model: SemanticVisualVLM,
benchmark_name: str,
dataset) -> dict:
"""Evaluate on standard VLM benchmarks."""
results = {
"accuracy": 0.0,
"hallucination_score": 0.0,
"semantic_consistency": 0.0
}
correct = 0
hallucination_count = 0
semantic_scores = []
for sample in dataset:
image = sample["image"]
question = sample["question"]
gold_answer = sample["answer"]
# Generate response
response = model.generate_response(
image,
question,
max_tokens=256
)
# Check correctness
if matches_answer(response, gold_answer):
correct += 1
# Check for hallucinations
if contains_hallucination(response, image):
hallucination_count += 1
# Semantic consistency score
semantic_score = compute_semantic_consistency(
image,
response,
model.semantic_tokenizer
)
semantic_scores.append(semantic_score)
results["accuracy"] = correct / len(dataset)
results["hallucination_score"] = 1.0 - (
hallucination_count / len(dataset)
)
results["semantic_consistency"] = sum(semantic_scores) / len(
semantic_scores
)
return results
Semantic Tokenization: VQ-SigLIP outperforms appearance-based tokenizers because high-level semantic structure matters more than pixel fidelity for understanding. Use semantic tokenizers rather than pixel reconstruction.
Joint Objective Balance: Equal weighting between visual and text losses works well in practice. If one modality dominates, adjust weights based on downstream task importance.
Training Data: Large-scale image-text pairs enable strong pre-training. Instruction-tuning datasets should cover diverse vision-language tasks (VQA, captioning, scene understanding).
Hallucination Reduction: The semantic visual supervision significantly reduces hallucinations (10-point improvement on HallusionBench). This is the primary benefit over text-only supervision.
When to Apply: Use ASVR when reducing hallucinations or improving visual understanding is critical, or when training on multimodal data with dense supervision.
ASVR achieves consistent 2-3% gains across 14 benchmarks by establishing explicit perceptual supervision alongside text objectives. The key insight is that semantic-level visual reconstruction (not pixel-level) provides the right inductive bias for understanding, leading to more robust and grounded vision-language models.