learning-visual-priors-llm-pretraining

name	learning-visual-priors-llm-pretraining
title	Learning Visual Priors Before Seeing: Optimized VLM Pretraining
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2509.26625
keywords	["VLM","pretraining","data-mixture","visual-understanding","foundation-models"]
description	Decompose visual priors into perception and reasoning components, each optimized by distinct data types: reasoning from code/math corpora, perception from diverse modality-rich sources. Use to construct efficient VLM pretraining pipelines balancing multimodal perception with reasoning capability.

Learning Visual Priors Before Seeing: Optimized VLM Pretraining

This research decomposes how Vision-Language Models develop visual understanding during text-only pretraining, identifying that perception and reasoning components arise from distinct data sources. By optimizing pretraining mixtures accordingly, practitioners can achieve competitive multimodal performance with reduced visual fine-tuning data.

Core Architecture

• Perception component: Emerges from diverse corpora (web text, books, scientific papers)
• Reasoning component: Scales with reasoning-focused data (code, mathematics)
• Mixture optimization: 60% reasoning + 15% visual-diverse content achieves balanced tradeoff
• Controlled experiments: 100+ systematic experiments identifying component origins

Implementation Steps

Design pretraining data mixture based on component analysis:

# Construct optimized data mixture for VLM pretraining
from vlm_mixture import DataMixture, PerceptionReasoningOptimizer

# Define component-aligned data sources
data_config = {
    "reasoning_content": {
        "code": 0.30,           # Python, JavaScript, SQL
        "mathematics": 0.20,    # Proofs, derivations, problem-solving
        "scientific": 0.10      # Academic papers with logical flow
    },
    "perception_content": {
        "web_text": 0.15,       # Diverse descriptions, narratives
        "books": 0.10,          # Rich contextual language
        "structured": 0.15      # Tables, lists, captions (visual context)
    }
}

mixture = DataMixture(
    config=data_config,
    total_tokens=1_000_000_000,  # 1T tokens
    adapter_architecture=
)