Skip to main content Inicio Creadores adu2021 skillxiv semantic-visual-reconstruction
semantic-visual-reconstruction Add explicit visual supervision to VLMs by training models to autoregressively reconstruct semantic image tokens, achieving 2-3% average gains and 10-point improvements on hallucination robustness.
Ir a la instalación Skills Marketplace Descubre y explora habilidades de IA creadas por la comunidad.
Instalar con Codex o Claude Copia este prompt, pégalo en Codex, Claude u otro asistente, y deja que revise la página de la skill y la instale por ti.
Mostrar detalles del prompt Un comando directo omite el prompt de revisión. Revisa el origen antes de ejecutarlo.
npx skills add https://github.com/ADu2021/skillXiv --skill semantic-visual-reconstructionEl comando permanece en una sola línea. Desplázate horizontalmente para revisarlo antes de copiarlo.
¿Prefieres una copia local? Descarga los archivos que SkillsMP tiene disponibles ahora.
Descargar Zip Descargando... Más de este repositorio meaningful-kebab-case-name Convert arXiv papers into ready-to-use agent skills using category-aware extraction. First classifies the paper into one or more of 11 research categories, then applies a specialized extraction pipeline for each category — because different types of papers produce different types of usable knowledge. A single paper can yield multiple skills if it spans categories. Use this skill whenever the user wants to turn a paper into a skill, extract practical techniques from research, build a skill library from papers, convert arXiv papers into reusable agent instructions, or batch-process multiple papers into skills. Also trigger when someone asks about extracting actionable knowledge from papers, making research practical for LLM agents, or systematically converting academic contributions into structured agent capabilities.
action-quantization-behavior-cloning Establish regret bounds for behavior cloning with discretized actions combining statistical error and quantization error terms. Prove smoothness requirements for safe quantizer design, show that learning-based quantizers fail these requirements, and propose model-based augmentation to reduce error dependence from H² to H.
adaptive-lora-personalized-ranks Dynamically allocate LoRA ranks per-layer during fine-tuning instead of using fixed uniform ranks. Learn optimal rank for each layer and subject via variational framework with discretized exponential distribution, reducing memory footprint while maintaining fidelity and text-alignment.
Ocupaciones relacionadas SOC
Basado en la clasificación ocupacional SOC
name semantic-visual-reconstruction title Autoregressive Semantic Visual Reconstruction Helps VLMs Understand Better version 0.0.2 engine skillxiv-v0.0.2-claude-opus-4.6 license MIT url https://arxiv.org/abs/2506.09040 keywords ["vision-language","semantic reconstruction","visual tokenization","multimodal supervision"] description Add explicit visual supervision to VLMs by training models to autoregressively reconstruct semantic image tokens, achieving 2-3% average gains and 10-point improvements on hallucination robustness.
Autoregressive Semantic Visual Reconstruction
Core Concept
Traditional Vision-Language Models (VLMs) apply supervision only to text outputs while leaving rich visual input unsupervised. ASVR (Autoregressive Semantic Visual Reconstruction) addresses this asymmetry by training models to predict both semantic visual tokens and text tokens within a unified framework. This establishes a "perceptual foundation for image understanding" that improves robustness and reduces hallucinations.
Architecture Overview
Visual tokenizer : VQ-SigLIP converts images to discrete semantic tokens capturing high-level featuresJoint training objective : Unified loss on both visual token and text token predictionTwo-stage training : Pre-training aligns visual representations, instruction tuning refines understandingSemantic > Appearance : High-level semantic information matters more than pixel-level reconstruction
Implementation
Step 1: Build Semantic Visual Tokenizer
Create tokenizer that captures high-level semantic features:
class SemanticVisualTokenizer :
def __init__ (self, model_name: str = "vq-siglip" ):
self .model_name = model_name
self .tokenizer = self ._load_semantic_tokenizer(model_name)
self .vocab_size = self .tokenizer.config.vocab_size
def _load_semantic_tokenizer (self, model_name: str ):
"""Load pretrained semantic visual tokenizer."""
from transformers import AutoModel
return AutoModel.from_pretrained(
f"semantic-tokenizers/{model_name} " ,
trust_remote_code=True
def
encode_image
self, image: torch.Tensor
"""Convert image to discrete semantic tokens."""
with
self
return
def
decode_tokens
self, tokens: torch.Tensor
"""Reconstruct image from semantic tokens."""
with
self
return
Step 2: Design Joint Training Objective Train model to predict both visual and text tokens:
class SemanticVisualVLM (torch.nn.Module):
def __init__ (self, text_model, vision_encoder,
semantic_tokenizer: SemanticVisualTokenizer ):
super ().__init__()
self .text_model = text_model
self .vision_encoder = vision_encoder
self .semantic_tokenizer = semantic_tokenizer
self .visual_head = torch.nn.Linear(
self .text_model.hidden_size,
semantic_tokenizer.vocab_size
)
self .text_head = torch.nn.Linear(
self .text_model.hidden_size,
self .text_model.vocab_size
)
def forward (self, image: torch.Tensor,
text_tokens: torch.Tensor,
visual_tokens: torch.Tensor ) -> dict :
"""Forward pass with both visual and text supervision."""
image_features = self .vision_encoder(image)
combined_input = self ._interleave_modalities(
image_features,
text_tokens
)
hidden_states = self .text_model(
combined_input,
output_hidden_states=True
).hidden_states
visual_logits = self .visual_head(hidden_states)
text_logits = self .text_head(hidden_states)
return {
"visual_logits" : visual_logits,
"text_logits" : text_logits,
"hidden_states" : hidden_states
}
def _interleave_modalities (self, image_features, text_tokens ):
"""Create combined input with visual and text modality markers."""
return torch.cat([image_features, text_tokens], dim=0 )
Step 3: Implement Two-Stage Training Stage 1: Pre-training on large-scale image-text data:
class PretrainingTrainer :
def __init__ (self, model: SemanticVisualVLM ):
self .model = model
self .optimizer = torch.optim.AdamW(
model.parameters(),
lr=1e-4
)
def compute_joint_loss (self, outputs: dict ,
visual_tokens: torch.Tensor,
text_tokens: torch.Tensor ) -> torch.Tensor:
"""Combined visual and text reconstruction loss."""
visual_logits = outputs["visual_logits" ]
visual_loss = torch.nn.functional.cross_entropy(
visual_logits.reshape(-1 , self .model.semantic_tokenizer.vocab_size),
visual_tokens.reshape(-1 )
)
text_logits = outputs["text_logits" ]
text_loss = torch.nn.functional.cross_entropy(
text_logits.reshape(-1 , self .model.text_model.vocab_size),
text_tokens.reshape(-1 )
)
total_loss = visual_loss + text_loss
return total_loss
def train_epoch (self, dataloader ):
"""Train one epoch on large-scale data."""
total_loss = 0.0
for batch in dataloader:
images = batch["image" ]
text_tokens = batch["text_tokens" ]
visual_tokens = self .model.semantic_tokenizer.encode_image(
images
)
outputs = self .model(
images,
text_tokens,
visual_tokens
)
loss = self .compute_joint_loss(
outputs,
visual_tokens,
text_tokens
)
self .optimizer.zero_grad()
loss.backward()
self .optimizer.step()
total_loss += loss.item()
return total_loss / len (dataloader)
Stage 2: Instruction tuning on diverse vision-language tasks:
class InstructionTuner :
def __init__ (self, pretrained_model: SemanticVisualVLM ):
self .model = pretrained_model
self .optimizer = torch.optim.AdamW(
self .model.parameters(),
lr=5e-5
)
def finetune_on_tasks (self, task_datasets: dict ,
num_epochs: int = 3 ):
"""Fine-tune on diverse instruction-following tasks."""
for epoch in range (num_epochs):
total_loss = 0.0
total_samples = 0
for task_name, dataset in task_datasets.items():
for batch in dataset:
images = batch["image" ]
instructions = batch["instruction" ]
target_responses = batch["response" ]
text_tokens = self .model.text_model.tokenize(
instructions
)
response_tokens = self .model.text_model.tokenize(
target_responses
)
visual_tokens = (
self .model.semantic_tokenizer.encode_image(images)
)
outputs = self .model(
images,
text_tokens,
visual_tokens
)
response_logits = outputs["text_logits" ][
len (text_tokens):
]
loss = torch.nn.functional.cross_entropy(
response_logits.reshape(
-1 ,
self .model.text_model.vocab_size
),
response_tokens.reshape(-1 )
)
self .optimizer.zero_grad()
loss.backward()
self .optimizer.step()
total_loss += loss.item()
total_samples += 1
avg_loss = total_loss / total_samples
print (f"Epoch {epoch} : Loss = {avg_loss:.4 f} " )
Step 4: Evaluate on Benchmarks Test improvements across multimodal understanding tasks:
def evaluate_model (model: SemanticVisualVLM,
benchmark_name: str ,
dataset ) -> dict :
"""Evaluate on standard VLM benchmarks."""
results = {
"accuracy" : 0.0 ,
"hallucination_score" : 0.0 ,
"semantic_consistency" : 0.0
}
correct = 0
hallucination_count = 0
semantic_scores = []
for sample in dataset:
image = sample["image" ]
question = sample["question" ]
gold_answer = sample["answer" ]
response = model.generate_response(
image,
question,
max_tokens=256
)
if matches_answer(response, gold_answer):
correct += 1
if contains_hallucination(response, image):
hallucination_count += 1
semantic_score = compute_semantic_consistency(
image,
response,
model.semantic_tokenizer
)
semantic_scores.append(semantic_score)
results["accuracy" ] = correct / len (dataset)
results["hallucination_score" ] = 1.0 - (
hallucination_count / len (dataset)
)
results["semantic_consistency" ] = sum (semantic_scores) / len (
semantic_scores
)
return results
Practical Guidance Semantic Tokenization : VQ-SigLIP outperforms appearance-based tokenizers because high-level semantic structure matters more than pixel fidelity for understanding. Use semantic tokenizers rather than pixel reconstruction.
Joint Objective Balance : Equal weighting between visual and text losses works well in practice. If one modality dominates, adjust weights based on downstream task importance.
Training Data : Large-scale image-text pairs enable strong pre-training. Instruction-tuning datasets should cover diverse vision-language tasks (VQA, captioning, scene understanding).
Hallucination Reduction : The semantic visual supervision significantly reduces hallucinations (10-point improvement on HallusionBench). This is the primary benefit over text-only supervision.
When to Apply : Use ASVR when reducing hallucinations or improving visual understanding is critical, or when training on multimodal data with dense supervision.
Reference ASVR achieves consistent 2-3% gains across 14 benchmarks by establishing explicit perceptual supervision alongside text objectives. The key insight is that semantic-level visual reconstruction (not pixel-level) provides the right inductive bias for understanding, leading to more robust and grounded vision-language models.
