lavit-visual-reasoning

name	lavit-visual-reasoning
title	LaViT: Aligning Latent Visual Thoughts for Multi-modal Reasoning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2601.10129
keywords	["vision-language","distillation","latent-alignment","multi-modal-reasoning","knowledge-transfer"]
description	Improves vision-language model distillation by aligning latent visual reasoning trajectories between teacher and student, enabling 3B parameter models to outperform larger open-source and proprietary systems with +16.9% gains on reasoning tasks.

Overview

Address the perception gap in knowledge distillation where students generate similar text to teachers but rely on different visual reasoning. Align latent visual thought trajectories between models, ensuring students genuinely understand images rather than taking shortcuts.

When to Use

• For compressing large vision-language models to smaller sizes
• When you want small models to perform as well as much larger models
• For improving multimodal reasoning capabilities in compact models
• When teaching models to ground reasoning in visual content

When NOT to Use

• For pure language or pure vision tasks without multimodal reasoning
• When computational overhead of trajectory alignment is unacceptable
• For real-time inference on extremely resource-constrained devices
• When larger models are not available as teachers

Key Technical Components

Perception Gap Detection

Identify when student models are taking language shortcuts.

# Detect language shortcut reliance
class PerceptionGapDetector:
    def diagnose_gap(self, teacher_output, student_output, image):
        """Detect if student relies on language vs visual understanding"""
        # Compare attention patterns
        teacher_visual_attention = teacher_output["visual_attention"]
        student_visual_attention = student_output["visual_attention"]

        # Compute attention divergence
        attention_kl = self.compute_kl_divergence(
            teacher_visual_attention,
            student_visual_attention
        )

        # If text outputs match but attention differs, perception gap exists
        text_match = teacher_output["text"] == student_output["text"]
        attention_mismatch = attention_kl > THRESHOLD

        gap_exists = text_match and attention_mismatch

        return {
            "gap_exists": gap_exists,
            "attention_divergence": attention_kl,
            "severity": .estimate_gap_severity(attention_kl)
        }

     ():
        
         kl_div < :
             
         kl_div < :
             
         kl_div < :
             
        :
             

     ():
        
        p = np.array(p) + 
        q = np.array(q) + 
         np.(p * (np.log(p) - np.log(q)))