adaptive-token-budget

name	adaptive-token-budget
description	Methods for adaptive/token-budgeted visual generation — token merging, token pruning, early exiting, and self-budgeting tokenization in diffusion/image generation models.

Adaptive Token-Budgeted Image Generation

Mental Model

A spectrum of techniques that reduce or adaptively allocate compute in visual generation models by treating tokens as a budgeted resource. At one end: training-free token merging that treats all tokens uniformly but merges similar ones. At the other end: learned token-budgeting where the model decides its own variable-length token count per input. In between are importance-weighted pruning, step-dependent early exiting, and caching strategies. All share the goal of spending compute where it matters — "minimum sufficient compute."

Coverage

Documented: AdaTok (self-budgeting tokenization), ToMe for Stable Diffusion (token merging), Importance-Based Token Merging, Cached Adaptive Token Merging, Early Exiting for Diffusion, No Compute Left Behind (adaptive masked diffusion sampling), Token Pruning for DiTs, Budget-Constrained Step-Level Caching. Not yet documented: Detailed technique modules in resources/techniques/. Last extended: 2026-06-22

Key Findings

AdaTok: Self-Budgeting Image Tokenization

• Location: docs/upstream/papers/lu2026adatok.md, arXiv:2606.07185
• What: A discrete 1D image tokenizer that learns variable-length token budgets per image (~118 avg tokens vs. fixed 256) in a single forward pass.
• Key ideas: Prioritized Representation Learning (nested tail masking + Multi-Head LoRA decoders), GRPO-based Adaptive Token Allocation, Dynamic Pareto Weighting.
• Why it matters: Direct instantiation of "capacity economy" and "subdivision governor" — a learned content-conditioned budget allocator.

Token Merging for Fast Stable Diffusion (ToMeSD)

• Location: arXiv:2303.17604
• What: Applies Token Merging (ToMe) to Stable Diffusion. Reduces tokens by up to 60% without training, speeds up generation by 2× and reduces memory by 5.6×.
• Key ideas: Diffusion-specific improvements over original ToMe; exploits natural redundancy in generated images.
• Why it matters: Training-free acceleration via token merging; stacks with xFormers.

Importance-Based Token Merging

• Location: arXiv:2411.16720
• What: Preserves high-information tokens during merging using importance scores from classifier-free guidance.
• Key ideas: Not all tokens are equal — semantic fidelity depends on keeping important tokens intact.
• Why it matters: Shows that content-aware budget allocation (not uniform merging) preserves quality.

Early Exiting Framework for Diffusion

• Location: arXiv:2408.05927 (ICML 2024)
• What: Time-dependent early-exit schedule: skips subsets of parameters based on the denoising time step.
• Key ideas: Compute needs vary across denoising steps; early steps benefit from full compute, later steps can exit early.
• Why it matters: Temporal adaptive compute — a complementary dimension to spatial token budgeting.

No Compute Left Behind (Masked Diffusion)

• Location: arXiv:2510.19990
• What: Multi-token entropy decoding (MED) — adaptive sampler that minimizes error by decoding positions in parallel based on conditional entropies.
• Key ideas: Any-order decoding in masked diffusion; early exit when model converges on answer; 2.7× fewer steps.

Token Pruning for DiTs

• Location: arXiv:2602.01609, arXiv:2605.14191
• What: Training-free token pruning for Diffusion Transformers using sensitivity analysis and influence metrics.
• Key ideas: Role asymmetry between context and target tokens; spatial-coherence guided pruning.
• Why it matters: Token reduction tailored to the in-context generation paradigm.

Critical Invariants

• Token merging and pruning are complementary to caching strategies — they address different bottlenecks (spatial redundancy vs. temporal redundancy across steps).
• All training-free methods degrade gracefully with higher reduction ratios; learned budget allocation (AdaTok) maintains quality better at high compression.
• Adaptive compute does not conflict with disposition layer insertion — they operate on different axes (token vs. layer dimension) and could be combined.

Sources

• arXiv API queries for "token merging diffusion", "adaptive token budget", "token pruning diffusion", "early exit diffusion", "compute budget diffusion"
• Individual arXiv abstract pages fetched via webfetch
• Papers indexed in references/papers.md under Architectural / Algorithmic