Skip to main content Accueil Créateurs adu2021 skillxiv semantic-visual-reconstruction
semantic-visual-reconstruction Add explicit visual supervision to VLMs by training models to autoregressively reconstruct semantic image tokens, achieving 2-3% average gains and 10-point improvements on hallucination robustness.
Aller à l'installation Skills Marketplace Découvrez et explorez les compétences IA créées par la communauté.
Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.
Afficher les détails du prompt Une commande directe contourne le prompt de vérification. Examinez la source avant de l'exécuter.
npx skills add https://github.com/ADu2021/skillXiv --skill semantic-visual-reconstructionLa commande reste sur une seule ligne. Faites défiler horizontalement pour la vérifier avant de la copier.
Vous préférez une copie locale ? Téléchargez les fichiers actuellement disponibles dans SkillsMP.
Télécharger Zip Téléchargement... Plus depuis ce dépôt meaningful-kebab-case-name Convert arXiv papers into ready-to-use agent skills using category-aware extraction. First classifies the paper into one or more of 11 research categories, then applies a specialized extraction pipeline for each category — because different types of papers produce different types of usable knowledge. A single paper can yield multiple skills if it spans categories. Use this skill whenever the user wants to turn a paper into a skill, extract practical techniques from research, build a skill library from papers, convert arXiv papers into reusable agent instructions, or batch-process multiple papers into skills. Also trigger when someone asks about extracting actionable knowledge from papers, making research practical for LLM agents, or systematically converting academic contributions into structured agent capabilities.
action-quantization-behavior-cloning Establish regret bounds for behavior cloning with discretized actions combining statistical error and quantization error terms. Prove smoothness requirements for safe quantizer design, show that learning-based quantizers fail these requirements, and propose model-based augmentation to reduce error dependence from H² to H.
adaptive-lora-personalized-ranks Dynamically allocate LoRA ranks per-layer during fine-tuning instead of using fixed uniform ranks. Learn optimal rank for each layer and subject via variational framework with discretized exponential distribution, reducing memory footprint while maintaining fidelity and text-alignment.
Métiers associés SOC
Basé sur la classification professionnelle SOC
name semantic-visual-reconstruction title Autoregressive Semantic Visual Reconstruction Helps VLMs Understand Better version 0.0.2 engine skillxiv-v0.0.2-claude-opus-4.6 license MIT url https://arxiv.org/abs/2506.09040 keywords ["vision-language","semantic reconstruction","visual tokenization","multimodal supervision"] description Add explicit visual supervision to VLMs by training models to autoregressively reconstruct semantic image tokens, achieving 2-3% average gains and 10-point improvements on hallucination robustness.
Autoregressive Semantic Visual Reconstruction
Core Concept
Traditional Vision-Language Models (VLMs) apply supervision only to text outputs while leaving rich visual input unsupervised. ASVR (Autoregressive Semantic Visual Reconstruction) addresses this asymmetry by training models to predict both semantic visual tokens and text tokens within a unified framework. This establishes a "perceptual foundation for image understanding" that improves robustness and reduces hallucinations.
Architecture Overview
Visual tokenizer : VQ-SigLIP converts images to discrete semantic tokens capturing high-level featuresJoint training objective : Unified loss on both visual token and text token predictionTwo-stage training : Pre-training aligns visual representations, instruction tuning refines understandingSemantic > Appearance : High-level semantic information matters more than pixel-level reconstruction
Implementation
Step 1: Build Semantic Visual Tokenizer
Create tokenizer that captures high-level semantic features:
class SemanticVisualTokenizer :
def __init__ (self, model_name: str = "vq-siglip" ):
self .model_name = model_name
self .tokenizer = self ._load_semantic_tokenizer(model_name)
self .vocab_size = self .tokenizer.config.vocab_size
def _load_semantic_tokenizer (self, model_name: str ):
"""Load pretrained semantic visual tokenizer."""
from transformers import AutoModel
return AutoModel.from_pretrained(
f"semantic-tokenizers/{model_name} " ,
trust_remote_code=True
def
encode_image
self, image: torch.Tensor
"""Convert image to discrete semantic tokens."""
with
self
return
def
decode_tokens
self, tokens: torch.Tensor
"""Reconstruct image from semantic tokens."""
with
self
return
Step 2: Design Joint Training Objective Train model to predict both visual and text tokens:
class SemanticVisualVLM (torch.nn.Module):
def __init__ (self, text_model, vision_encoder,
semantic_tokenizer: SemanticVisualTokenizer ):
super ().__init__()
self .text_model = text_model
self .vision_encoder = vision_encoder
self .semantic_tokenizer = semantic_tokenizer
self .visual_head = torch.nn.Linear(
self .text_model.hidden_size,
semantic_tokenizer.vocab_size
)
self .text_head = torch.nn.Linear(
self .text_model.hidden_size,
self .text_model.vocab_size
)
def forward (self, image: torch.Tensor,
text_tokens: torch.Tensor,
visual_tokens: torch.Tensor ) -> dict :
"""Forward pass with both visual and text supervision."""
image_features = self .vision_encoder(image)
combined_input = self ._interleave_modalities(
image_features,
text_tokens
)
hidden_states = self .text_model(
combined_input,
output_hidden_states=True
).hidden_states
visual_logits = self .visual_head(hidden_states)
text_logits = self .text_head(hidden_states)
return {
"visual_logits" : visual_logits,
"text_logits" : text_logits,
"hidden_states" : hidden_states
}
def _interleave_modalities (self, image_features, text_tokens ):
"""Create combined input with visual and text modality markers."""
return torch.cat([image_features, text_tokens], dim=0 )
Step 3: Implement Two-Stage Training Stage 1: Pre-training on large-scale image-text data:
class PretrainingTrainer :
def __init__ (self, model: SemanticVisualVLM ):
self .model = model
self .optimizer = torch.optim.AdamW(
model.parameters(),
lr=1e-4
)
def compute_joint_loss (self, outputs: dict ,
visual_tokens: torch.Tensor,
text_tokens: torch.Tensor ) -> torch.Tensor:
"""Combined visual and text reconstruction loss."""
visual_logits = outputs["visual_logits" ]
visual_loss = torch.nn.functional.cross_entropy(
visual_logits.reshape(-1 , self .model.semantic_tokenizer.vocab_size),
visual_tokens.reshape(-1 )
)
text_logits = outputs["text_logits" ]
text_loss = torch.nn.functional.cross_entropy(
text_logits.reshape(-1 , self .model.text_model.vocab_size),
text_tokens.reshape(-1 )
)
total_loss = visual_loss + text_loss
return total_loss
def train_epoch (self, dataloader ):
"""Train one epoch on large-scale data."""
total_loss = 0.0
for batch in dataloader:
images = batch["image" ]
text_tokens = batch["text_tokens" ]
visual_tokens = self .model.semantic_tokenizer.encode_image(
images
)
outputs = self .model(
images,
text_tokens,
visual_tokens
)
loss = self .compute_joint_loss(
outputs,
visual_tokens,
text_tokens
)
self .optimizer.zero_grad()
loss.backward()
self .optimizer.step()
total_loss += loss.item()
return total_loss / len (dataloader)
Stage 2: Instruction tuning on diverse vision-language tasks:
class InstructionTuner :
def __init__ (self, pretrained_model: SemanticVisualVLM ):
self .model = pretrained_model
self .optimizer = torch.optim.AdamW(
self .model.parameters(),
lr=5e-5
)
def finetune_on_tasks (self, task_datasets: dict ,
num_epochs: int = 3 ):
"""Fine-tune on diverse instruction-following tasks."""
for epoch in range (num_epochs):
total_loss = 0.0
total_samples = 0
for task_name, dataset in task_datasets.items():
for batch in dataset:
images = batch["image" ]
instructions = batch["instruction" ]
target_responses = batch["response" ]
text_tokens = self .model.text_model.tokenize(
instructions
)
response_tokens = self .model.text_model.tokenize(
target_responses
)
visual_tokens = (
self .model.semantic_tokenizer.encode_image(images)
)
outputs = self .model(
images,
text_tokens,
visual_tokens
)
response_logits = outputs["text_logits" ][
len (text_tokens):
]
loss = torch.nn.functional.cross_entropy(
response_logits.reshape(
-1 ,
self .model.text_model.vocab_size
),
response_tokens.reshape(-1 )
)
self .optimizer.zero_grad()
loss.backward()
self .optimizer.step()
total_loss += loss.item()
total_samples += 1
avg_loss = total_loss / total_samples
print (f"Epoch {epoch} : Loss = {avg_loss:.4 f} " )
Step 4: Evaluate on Benchmarks Test improvements across multimodal understanding tasks:
def evaluate_model (model: SemanticVisualVLM,
benchmark_name: str ,
dataset ) -> dict :
"""Evaluate on standard VLM benchmarks."""
results = {
"accuracy" : 0.0 ,
"hallucination_score" : 0.0 ,
"semantic_consistency" : 0.0
}
correct = 0
hallucination_count = 0
semantic_scores = []
for sample in dataset:
image = sample["image" ]
question = sample["question" ]
gold_answer = sample["answer" ]
response = model.generate_response(
image,
question,
max_tokens=256
)
if matches_answer(response, gold_answer):
correct += 1
if contains_hallucination(response, image):
hallucination_count += 1
semantic_score = compute_semantic_consistency(
image,
response,
model.semantic_tokenizer
)
semantic_scores.append(semantic_score)
results["accuracy" ] = correct / len (dataset)
results["hallucination_score" ] = 1.0 - (
hallucination_count / len (dataset)
)
results["semantic_consistency" ] = sum (semantic_scores) / len (
semantic_scores
)
return results
Practical Guidance Semantic Tokenization : VQ-SigLIP outperforms appearance-based tokenizers because high-level semantic structure matters more than pixel fidelity for understanding. Use semantic tokenizers rather than pixel reconstruction.
Joint Objective Balance : Equal weighting between visual and text losses works well in practice. If one modality dominates, adjust weights based on downstream task importance.
Training Data : Large-scale image-text pairs enable strong pre-training. Instruction-tuning datasets should cover diverse vision-language tasks (VQA, captioning, scene understanding).
Hallucination Reduction : The semantic visual supervision significantly reduces hallucinations (10-point improvement on HallusionBench). This is the primary benefit over text-only supervision.
When to Apply : Use ASVR when reducing hallucinations or improving visual understanding is critical, or when training on multimodal data with dense supervision.
Reference ASVR achieves consistent 2-3% gains across 14 benchmarks by establishing explicit perceptual supervision alongside text objectives. The key insight is that semantic-level visual reconstruction (not pixel-level) provides the right inductive bias for understanding, leading to more robust and grounded vision-language models.
