Skip to main content الرئيسية المنشئون adu2021 skillxiv semantic-visual-reconstruction
semantic-visual-reconstruction Add explicit visual supervision to VLMs by training models to autoregressively reconstruct semantic image tokens, achieving 2-3% average gains and 10-point improvements on hallucination robustness.
الانتقال إلى التثبيت سوق المهارات اكتشف واستكشف مهارات الذكاء الاصطناعي التي بناها المجتمع.
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
عرض تفاصيل Prompt يتجاوز الأمر المباشر Prompt المخصّص للمراجعة. افحص المصدر قبل تشغيله.
npx skills add https://github.com/ADu2021/skillXiv --skill semantic-visual-reconstructionيبقى الأمر في سطر واحد. مرّر أفقيًا لمراجعته كاملًا قبل النسخ.
تفضّل نسخة محلية؟ نزّل الملفات المتاحة حاليًا لدى SkillsMP.
تحميل Zip جاري التحميل... name semantic-visual-reconstruction title Autoregressive Semantic Visual Reconstruction Helps VLMs Understand Better version 0.0.2 engine skillxiv-v0.0.2-claude-opus-4.6 license MIT url https://arxiv.org/abs/2506.09040 keywords ["vision-language","semantic reconstruction","visual tokenization","multimodal supervision"] description Add explicit visual supervision to VLMs by training models to autoregressively reconstruct semantic image tokens, achieving 2-3% average gains and 10-point improvements on hallucination robustness.
Autoregressive Semantic Visual Reconstruction
Core Concept
Traditional Vision-Language Models (VLMs) apply supervision only to text outputs while leaving rich visual input unsupervised. ASVR (Autoregressive Semantic Visual Reconstruction) addresses this asymmetry by training models to predict both semantic visual tokens and text tokens within a unified framework. This establishes a "perceptual foundation for image understanding" that improves robustness and reduces hallucinations.
Architecture Overview
Visual tokenizer : VQ-SigLIP converts images to discrete semantic tokens capturing high-level featuresJoint training objective : Unified loss on both visual token and text token predictionTwo-stage training : Pre-training aligns visual representations, instruction tuning refines understandingSemantic > Appearance : High-level semantic information matters more than pixel-level reconstruction
Implementation
Step 1: Build Semantic Visual Tokenizer
Create tokenizer that captures high-level semantic features:
class SemanticVisualTokenizer :
def __init__ (self, model_name: str = "vq-siglip" ):
self .model_name = model_name
self .tokenizer = self ._load_semantic_tokenizer(model_name)
self .vocab_size = self .tokenizer.config.vocab_size
def _load_semantic_tokenizer (self, model_name: str ):
"""Load pretrained semantic visual tokenizer."""
from transformers import AutoModel
return AutoModel.from_pretrained(
f"semantic-tokenizers/{model_name} " ,
trust_remote_code=True
def
encode_image
self, image: torch.Tensor
"""Convert image to discrete semantic tokens."""
with
self
return
def
decode_tokens
self, tokens: torch.Tensor
"""Reconstruct image from semantic tokens."""
with
self
return
Step 2: Design Joint Training Objective Train model to predict both visual and text tokens:
class SemanticVisualVLM (torch.nn.Module):
def __init__ (self, text_model, vision_encoder,
semantic_tokenizer: SemanticVisualTokenizer ):
super ().__init__()
self .text_model = text_model
self .vision_encoder = vision_encoder
self .semantic_tokenizer = semantic_tokenizer
self .visual_head = torch.nn.Linear(
self .text_model.hidden_size,
semantic_tokenizer.vocab_size
)
self .text_head = torch.nn.Linear(
self .text_model.hidden_size,
self .text_model.vocab_size
)
def forward (self, image: torch.Tensor,
text_tokens: torch.Tensor,
visual_tokens: torch.Tensor ) -> dict :
"""Forward pass with both visual and text supervision."""
image_features = self .vision_encoder(image)
combined_input = self ._interleave_modalities(
image_features,
text_tokens
)
hidden_states = self .text_model(
combined_input,
output_hidden_states=True
).hidden_states
visual_logits = self .visual_head(hidden_states)
text_logits = self .text_head(hidden_states)
return {
"visual_logits" : visual_logits,
"text_logits" : text_logits,
"hidden_states" : hidden_states
}
def _interleave_modalities (self, image_features, text_tokens ):
"""Create combined input with visual and text modality markers."""
return torch.cat([image_features, text_tokens], dim=0 )
Step 3: Implement Two-Stage Training Stage 1: Pre-training on large-scale image-text data:
class PretrainingTrainer :
def __init__ (self, model: SemanticVisualVLM ):
self .model = model
self .optimizer = torch.optim.AdamW(
model.parameters(),
lr=1e-4
)
def compute_joint_loss (self, outputs: dict ,
visual_tokens: torch.Tensor,
text_tokens: torch.Tensor ) -> torch.Tensor:
"""Combined visual and text reconstruction loss."""
visual_logits = outputs["visual_logits" ]
visual_loss = torch.nn.functional.cross_entropy(
visual_logits.reshape(-1 , self .model.semantic_tokenizer.vocab_size),
visual_tokens.reshape(-1 )
)
text_logits = outputs["text_logits" ]
text_loss = torch.nn.functional.cross_entropy(
text_logits.reshape(-1 , self .model.text_model.vocab_size),
text_tokens.reshape(-1 )
)
total_loss = visual_loss + text_loss
return total_loss
def train_epoch (self, dataloader ):
"""Train one epoch on large-scale data."""
total_loss = 0.0
for batch in dataloader:
images = batch["image" ]
text_tokens = batch["text_tokens" ]
visual_tokens = self .model.semantic_tokenizer.encode_image(
images
)
outputs = self .model(
images,
text_tokens,
visual_tokens
)
loss = self .compute_joint_loss(
outputs,
visual_tokens,
text_tokens
)
self .optimizer.zero_grad()
loss.backward()
self .optimizer.step()
total_loss += loss.item()
return total_loss / len (dataloader)
Stage 2: Instruction tuning on diverse vision-language tasks:
class InstructionTuner :
def __init__ (self, pretrained_model: SemanticVisualVLM ):
self .model = pretrained_model
self .optimizer = torch.optim.AdamW(
self .model.parameters(),
lr=5e-5
)
def finetune_on_tasks (self, task_datasets: dict ,
num_epochs: int = 3 ):
"""Fine-tune on diverse instruction-following tasks."""
for epoch in range (num_epochs):
total_loss = 0.0
total_samples = 0
for task_name, dataset in task_datasets.items():
for batch in dataset:
images = batch["image" ]
instructions = batch["instruction" ]
target_responses = batch["response" ]
text_tokens = self .model.text_model.tokenize(
instructions
)
response_tokens = self .model.text_model.tokenize(
target_responses
)
visual_tokens = (
self .model.semantic_tokenizer.encode_image(images)
)
outputs = self .model(
images,
text_tokens,
visual_tokens
)
response_logits = outputs["text_logits" ][
len (text_tokens):
]
loss = torch.nn.functional.cross_entropy(
response_logits.reshape(
-1 ,
self .model.text_model.vocab_size
),
response_tokens.reshape(-1 )
)
self .optimizer.zero_grad()
loss.backward()
self .optimizer.step()
total_loss += loss.item()
total_samples += 1
avg_loss = total_loss / total_samples
print (f"Epoch {epoch} : Loss = {avg_loss:.4 f} " )
Step 4: Evaluate on Benchmarks Test improvements across multimodal understanding tasks:
def evaluate_model (model: SemanticVisualVLM,
benchmark_name: str ,
dataset ) -> dict :
"""Evaluate on standard VLM benchmarks."""
results = {
"accuracy" : 0.0 ,
"hallucination_score" : 0.0 ,
"semantic_consistency" : 0.0
}
correct = 0
hallucination_count = 0
semantic_scores = []
for sample in dataset:
image = sample["image" ]
question = sample["question" ]
gold_answer = sample["answer" ]
response = model.generate_response(
image,
question,
max_tokens=256
)
if matches_answer(response, gold_answer):
correct += 1
if contains_hallucination(response, image):
hallucination_count += 1
semantic_score = compute_semantic_consistency(
image,
response,
model.semantic_tokenizer
)
semantic_scores.append(semantic_score)
results["accuracy" ] = correct / len (dataset)
results["hallucination_score" ] = 1.0 - (
hallucination_count / len (dataset)
)
results["semantic_consistency" ] = sum (semantic_scores) / len (
semantic_scores
)
return results
Practical Guidance Semantic Tokenization : VQ-SigLIP outperforms appearance-based tokenizers because high-level semantic structure matters more than pixel fidelity for understanding. Use semantic tokenizers rather than pixel reconstruction.
Joint Objective Balance : Equal weighting between visual and text losses works well in practice. If one modality dominates, adjust weights based on downstream task importance.
Training Data : Large-scale image-text pairs enable strong pre-training. Instruction-tuning datasets should cover diverse vision-language tasks (VQA, captioning, scene understanding).
Hallucination Reduction : The semantic visual supervision significantly reduces hallucinations (10-point improvement on HallusionBench). This is the primary benefit over text-only supervision.
When to Apply : Use ASVR when reducing hallucinations or improving visual understanding is critical, or when training on multimodal data with dense supervision.
Reference ASVR achieves consistent 2-3% gains across 14 benchmarks by establishing explicit perceptual supervision alongside text objectives. The key insight is that semantic-level visual reconstruction (not pixel-level) provides the right inductive bias for understanding, leading to more robust and grounded vision-language models.
المهن ذات الصلة SOC
استنادا إلى تصنيف SOC المهني
