byte-modeling-efficiency-gap

# Byte Modeling Efficiency Gap ## Trigger Words byte modeling efficiency, byte-level language model, masked diffusion model efficiency, context fragility, modality-agnostic generation, subword tokenization alternatives ## Core Idea Modern language models rely on subword tokenization and autoregressive ordering as design priors. Byte-level modeling bypasses static token vocabularies, and masked diffusion modeling (MDM) enables parallel non-sequential generation. Their intersection represents a fully end-to-end modality-agnostic generative prototype, but removing structural priors incurs significant computational cost. ## Key Findings ### 1. Compute-Matched Scaling Study - Performance penalty of byte modeling is NOT uniform across scale - Scaling overhead of byte modeling is worse for MDM than for AR - The gap widens at larger scales ### 2. Context Fragility Hypothesis - AR's stable causal history allows models to naturally rediscover subword patterns - MDM objective destroys local contiguity required to efficiently resolve semantics from raw bytes - MDM's parallel generation loses the sequential structure that aids byte-level pattern discovery ### 3. Permutation Experiment Results - Controlled experiments suggest context ordering matters differently for MDM vs AR - MDM is more sensitive to loss of local contiguity in byte regime ## Recommendations for Future Designs - Modality-agnostic byte models must incorporate alternative structural biases - Need new architectural inductive biases to maintain viable scaling trajectories - Cannot simply remove tokenization without compensating with other structural guidance ## Design Implications - Byte-level + MDM combination requires explicit structural bias injection - Possible directions: explicit locality modules, hierarchical byte grouping, or learned substructure discovery - AR byte modeling is more viable than MDM byte modeling at current scale ## Source arXiv: 2605.12928v1 - "The Efficiency Gap in Byte Modeling"