Enable multimodal models to dynamically switch between text and vision reasoning modes, allocating computation based on perceived difficulty and image resolution, achieving strong performance on both vision-dense and text-heavy benchmarks.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Enable multimodal models to dynamically switch between text and vision reasoning modes, allocating computation based on perceived difficulty and image resolution, achieving strong performance on both vision-dense and text-heavy benchmarks.
SwimBird: Eliciting Switchable Reasoning Mode in Hybrid Autoregressive MLLMs
Problem Context
Fixed reasoning patterns in multimodal systems mismatch question types with computation modes. Text-heavy problems forced to use visual thinking degrade logic quality; vision-dense tasks forced into text-only reasoning lose spatial detail. Models need to adapt their reasoning modality per query rather than commit to a single approach.
Core Concept
SwimBird combines [hybrid autoregressive modeling, dynamic latent budgets, multi-mode dataset curation] to enable query-adaptive mode selection. The model predicts reasoning difficulty and image resolution, then allocates variable continuous tokens (visual thoughts) dynamically, switching seamlessly between text-only and vision-rich paths.
Architecture Overview
Hybrid autoregressive: Next-token prediction for text, next-embedding prediction for visual tokens
Dynamic allocation: Variable visual-thought budget based on difficulty scores and resolution
Mode switching: Special delimiters enable flexible mode transitions
Dataset curation: SwimBird-SFT-92K categorizes data by reasoning pattern using pass@8 scoring
Resolution awareness: Adapt token allocation to image resolution
Implementation
Step 1: Implement hybrid autoregressive model
Create dual-mode generation supporting both discrete tokens and continuous embeddings.
Implement training that respects the best reasoning mode for each example.
# Mode-aware trainingdeftrain_swimbird(
model, train_loader, optimizer, device='cuda',
lambda_mode=0.1):
"""
Train SwimBird with mode-aware supervision.
"""
model = model.to(device)
criterion_token = torch.nn.CrossEntropyLoss()
criterion_vision = torch.nn.MSELoss()
for epoch inrange(3):
total_loss = 0.0for batch in train_loader:
question = batch['question']
image = batch.get('image')
answer = batch['answer']
best_mode = batch['best_mode']
# Forward pass
token_logits, vision_preds, mode_logits = model(
input_ids=question,
vision_embeddings=image,
mode_mask=None# Will be determined dynamically
)
# Token prediction loss
answer_ids = model.tokenize(answer)
token_loss = criterion_token(token_logits, answer_ids)
# Mode supervision loss
mode_labels = torch.tensor([
0if m == 'text_only'else1for m in best_mode
]).to(device)
mode_loss = torch.nn.functional.cross_entropy(
mode_logits[:, 0], mode_labels
)
# Vision embedding loss (if applicable)if image isnotNone:
vision_loss = criterion_vision(vision_preds, image)
else:
vision_loss = 0.0# Combined loss
total = token_loss + lambda_mode * mode_loss + 0.5 * vision_loss
optimizer.zero_grad()
total.backward()
optimizer.step()
total_loss += total.item()
print(f"Epoch {epoch + 1}: Loss={total_loss / len(train_loader):.4f}")
Step 5: Inference with dynamic mode selection
Run inference with adaptive mode switching.
# Adaptive inferencedefgenerate_with_adaptive_reasoning(
model, question, image=None, max_tokens=200):
"""
Generate response with dynamically selected reasoning mode.
"""
difficulty_detector = DifficultyDetector(model)
# Detect difficulty and resolution
difficulty = difficulty_detector.detect_difficulty(question, image)
resolution = difficulty_detector.detect_image_resolution(image) if image else0# Allocate vision budget
vision_budget = difficulty_detector.allocate_vision_budget(
difficulty, resolution
)
# Generate with allocated budget
response = []
current_mode = 'text'# Start with text
vision_tokens_used = 0for step inrange(max_tokens):
# Decide mode at each stepif vision_tokens_used < vision_budget and image isnotNone:
# Can use vision tokens
mode_logits = model.mode_logits(model_hidden_state)
mode_probs = torch.softmax(mode_logits, dim=-1)
current_mode = 'vision'if mode_probs[1] > 0.5else'text'else:
current_mode = 'text'# Generate next token/embedding based on modeif current_mode == 'text':
next_token_logits = model.token_head(model_hidden_state)
next_token = next_token_logits.argmax(dim=-1)
response.append(model.decode_token(next_token))
else:
next_vision_emb = model.vision_embedding_head(model_hidden_state)
response.append(f"[VISION_TOKEN_{vision_tokens_used}]")
vision_tokens_used += 1# Early stoppingif next_token == model.eos_token_id:
breakreturn' '.join(response)
Practical Guidance
When to use: Multimodal benchmarks with mixed question types (vision-heavy and text-heavy). Most beneficial for question-answering, document understanding, scientific reasoning.
Hyperparameters:
Max vision tokens: 64-256 (balance visual detail vs. text length)
