Enable multimodal models to dynamically switch between text and vision reasoning modes, allocating computation based on perceived difficulty and image resolution, achieving strong performance on both vision-dense and text-heavy benchmarks.
Enable multimodal models to dynamically switch between text and vision reasoning modes, allocating computation based on perceived difficulty and image resolution, achieving strong performance on both vision-dense and text-heavy benchmarks.
SwimBird: Eliciting Switchable Reasoning Mode in Hybrid Autoregressive MLLMs
Problem Context
Fixed reasoning patterns in multimodal systems mismatch question types with computation modes. Text-heavy problems forced to use visual thinking degrade logic quality; vision-dense tasks forced into text-only reasoning lose spatial detail. Models need to adapt their reasoning modality per query rather than commit to a single approach.
Core Concept
SwimBird combines [hybrid autoregressive modeling, dynamic latent budgets, multi-mode dataset curation] to enable query-adaptive mode selection. The model predicts reasoning difficulty and image resolution, then allocates variable continuous tokens (visual thoughts) dynamically, switching seamlessly between text-only and vision-rich paths.
Architecture Overview
Hybrid autoregressive: Next-token prediction for text, next-embedding prediction for visual tokens
Dynamic allocation: Variable visual-thought budget based on difficulty scores and resolution
Mode switching: Special delimiters enable flexible mode transitions
Dataset curation: SwimBird-SFT-92K categorizes data by reasoning pattern using pass@8 scoring
Resolution awareness: Adapt token allocation to image resolution
Implementation
Step 1: Implement hybrid autoregressive model
Create dual-mode generation supporting both discrete tokens and continuous embeddings.
Implement training that respects the best reasoning mode for each example.
# Mode-aware trainingdeftrain_swimbird(
model, train_loader, optimizer, device='cuda',
lambda_mode=0.1):
"""
Train SwimBird with mode-aware supervision.
"""
model = model.to(device)
criterion_token = torch.nn.CrossEntropyLoss()
criterion_vision = torch.nn.MSELoss()
for epoch inrange(3):
total_loss = 0.0for batch in train_loader:
question = batch['question']
image = batch.get('image')
answer = batch['answer']
best_mode = batch['best_mode']
# Forward pass
token_logits, vision_preds, mode_logits = model(
input_ids=question,
vision_embeddings=image,
mode_mask=None# Will be determined dynamically
)
# Token prediction loss
answer_ids = model.tokenize(answer)
token_loss = criterion_token(token_logits, answer_ids)
# Mode supervision loss
mode_labels = torch.tensor([
0if m == 'text_only'else1for m in best_mode
]).to(device)
mode_loss = torch.nn.functional.cross_entropy(
mode_logits[:, 0], mode_labels
)
# Vision embedding loss (if applicable)if image isnotNone:
vision_loss = criterion_vision(vision_preds, image)
else:
vision_loss = 0.0# Combined loss
total = token_loss + lambda_mode * mode_loss + 0.5 * vision_loss
optimizer.zero_grad()
total.backward()
optimizer.step()
total_loss += total.item()
print(f"Epoch {epoch + 1}: Loss={total_loss / len(train_loader):.4f}")
Step 5: Inference with dynamic mode selection
Run inference with adaptive mode switching.
# Adaptive inferencedefgenerate_with_adaptive_reasoning(
model, question, image=None, max_tokens=200):
"""
Generate response with dynamically selected reasoning mode.
"""
difficulty_detector = DifficultyDetector(model)
# Detect difficulty and resolution
difficulty = difficulty_detector.detect_difficulty(question, image)
resolution = difficulty_detector.detect_image_resolution(image) if image else0# Allocate vision budget
vision_budget = difficulty_detector.allocate_vision_budget(
difficulty, resolution
)
# Generate with allocated budget
response = []
current_mode = 'text'# Start with text
vision_tokens_used = 0for step inrange(max_tokens):
# Decide mode at each stepif vision_tokens_used < vision_budget and image isnotNone:
# Can use vision tokens
mode_logits = model.mode_logits(model_hidden_state)
mode_probs = torch.softmax(mode_logits, dim=-1)
current_mode = 'vision'if mode_probs[1] > 0.5else'text'else:
current_mode = 'text'# Generate next token/embedding based on modeif current_mode == 'text':
next_token_logits = model.token_head(model_hidden_state)
next_token = next_token_logits.argmax(dim=-1)
response.append(model.decode_token(next_token))
else:
next_vision_emb = model.vision_embedding_head(model_hidden_state)
response.append(f"[VISION_TOKEN_{vision_tokens_used}]")
vision_tokens_used += 1# Early stoppingif next_token == model.eos_token_id:
breakreturn' '.join(response)
Practical Guidance
When to use: Multimodal benchmarks with mixed question types (vision-heavy and text-heavy). Most beneficial for question-answering, document understanding, scientific reasoning.
Hyperparameters:
Max vision tokens: 64-256 (balance visual detail vs. text length)
