Skip to main content Home Creators adu2021 skillxiv mm-helix-reasoning
mm-helix-reasoning Train multimodal models for long-chain reflective reasoning (iterative thinking, backtracking) using Adaptive Hybrid Policy Optimization. Trigger: improve VLM performance on tasks requiring iterative refinement and error correction.
Jump to install Skills Marketplace Discover and explore AI skills built by the community.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Show prompt details A direct command skips the review prompt. Inspect the source before running it.
npx skills add https://github.com/ADu2021/skillXiv --skill mm-helix-reasoningThe command stays on one line. Scroll horizontally to inspect it before copying.
Prefer a local copy? Download the files currently available to SkillsMP.
Download Zip Downloading... More from this repository
Related occupations SOC
Based on SOC occupation classification
name mm-helix-reasoning title MM-HELIX: Multimodal Long-Chain Reflective Reasoning via AHPO version 0.0.2 engine skillxiv-v0.0.2-claude-opus-4.6 license MIT url https://arxiv.org/abs/2510.08540 keywords ["multimodal-reasoning","reflective-thinking","reinforcement-learning","backtracking","adaptive-training"] description Train multimodal models for long-chain reflective reasoning (iterative thinking, backtracking) using Adaptive Hybrid Policy Optimization. Trigger: improve VLM performance on tasks requiring iterative refinement and error correction.
MM-HELIX: Multimodal Reflective Reasoning with AHPO
Core Concept
MM-HELIX addresses a critical gap in multimodal LLMs: they struggle with long-chain reflective reasoning that requires iterative thinking and backtracking. The paper introduces three components: a benchmark (MM-HELIX) with 1,260 samples requiring iterative thinking, a large-scale training dataset (MM-HELIX-100K), and Adaptive Hybrid Policy Optimization (AHPO) that dynamically unifies offline supervision with online RL in a single training stage, achieving 18.6% accuracy improvement.
The key insight: Reflective reasoning requires dynamic training that switches between supervised learning on correct traces and online RL refinement—not static offline or online training alone.
Architecture Overview
Long-Chain Reasoning : Multi-step thinking with iterative refinementBacktracking Capability : Ability to detect and correct reasoning errorsMM-HELIX-100K Dataset : 100K high-quality reasoning traces with iterative patternsAdaptive Hybrid Policy Optimization : Unified offline+online training in single stageCatastrophic Forgetting Prevention : Dynamic balancing prevents mode collapse
Implementation Steps
1. Understand Long-Chain Reflective Reasoning Structure
Define what constitutes reflective reasoning in multimodal contexts.
class ReflectiveReasoningStructure :
"""
Represent multi-step reasoning with backtracking.
"""
def __init__ (self ):
pass
@staticmethod
def extract_reasoning_steps (trace ):
"""
Parse reasoning into steps with branch points.
"""
steps = []
current_path = []
lines = trace.split('\n' )
for line in lines:
if 'THOUGHT:' in line:
current_path.append({
"type" : "thought" ,
"content"
'THOUGHT:'
1
elif
'BACKTRACK:'
in
"path"
"backtrack_reason"
'BACKTRACK:'
1
elif
'CONCLUSION:'
in
"path"
"conclusion"
'CONCLUSION:'
1
return
@staticmethod
def
score_reasoning_quality
trace, ground_truth
"""
Evaluate reasoning quality: correctness, efficiency, iterativeness.
"""
1.0
if
else
0.0
sum
1
for
in
if
"backtrack_reason"
in
min
5.0
1.0
len
1.0
1.0
500
0.7
0.2
0.1
return
"quality_score"
"correctness"
"iterativeness"
"efficiency"
2. Implement Step-Elicited Response Generation (SERG) Create high-quality reasoning traces with explicit step-by-step generation.
class StepElicitedResponseGenerator :
"""
Systematically generate reasoning traces with explicit steps.
"""
def __init__ (self, model ):
self .model = model
def generate_with_steps (self, problem, image=None , max_iterations=5 ):
"""
Generate reasoning by eliciting explicit steps.
Args:
problem: Problem statement
image: Multimodal input (optional)
max_iterations: Max refinement iterations
Returns:
Multi-step reasoning trace
"""
trace = f"Problem: {problem} \n\n"
if image:
trace += f"[Image analysis]\n"
iteration = 0
current_solution = None
while iteration < max_iterations:
thought_prompt = (
trace +
f"\n[Step {iteration + 1 } ]\n"
f"Thinking: "
)
thought = self .model.generate(
thought_prompt,
max_tokens=150 ,
temperature=0.7
)
trace += f"THOUGHT: {thought} \n"
conclusion_prompt = (
trace +
f"Conclusion from this step: "
)
conclusion = self .model.generate(
conclusion_prompt,
max_tokens=100 ,
temperature=0.3
)
trace += f"CONCLUSION: {conclusion} \n"
current_solution = conclusion
evaluation_prompt = (
trace +
f"Is this solution correct and complete? "
f"[Yes/No]. If No, what needs to be reconsidered? "
)
evaluation = self .model.generate(
evaluation_prompt,
max_tokens=50 ,
temperature=0.5
)
if "yes" in evaluation.lower():
trace += "FINAL: Solution is correct.\n"
break
else :
backtrack_reason = evaluation.split("No" )[-1 ].strip()
trace += f"BACKTRACK: {backtrack_reason} \n"
trace += "[Reconsidering...]\n"
iteration += 1
return trace
def generate_dataset_with_serg (self, problems, images, num_samples=100000 ):
"""
Systematically generate 100K reasoning traces.
"""
dataset = []
for idx, (problem, image) in enumerate (zip (problems, images)):
trace = self .generate_with_steps(problem, image)
quality = ReflectiveReasoningStructure.score_reasoning_quality(
trace,
ground_truth=extract_answer(problem)
)
example = {
"problem" : problem,
"image" : image,
"reasoning" : trace,
"quality_score" : quality["quality_score" ]
}
dataset.append(example)
if (idx + 1 ) % 10000 == 0 :
print (f"Generated {idx + 1 } examples" )
if len (dataset) >= num_samples:
break
return dataset
3. Implement Adaptive Hybrid Policy Optimization (AHPO) Dynamically blend offline supervision with online RL in a single unified stage.
class AdaptiveHybridPolicyOptimization :
"""
Unified training that adapts between supervised and online RL.
"""
def __init__ (self, model ):
self .model = model
def compute_ahpo_loss (self, batch, step, total_steps ):
"""
Compute loss that adapts between offline and online optimization.
Args:
batch: Training batch with problems, images, reasoning traces
step: Current training step
total_steps: Total steps in training
Returns:
Scalar loss value
"""
total_loss = 0
batch_size = len (batch)
offline_weight = max (0.7 - (step / total_steps) * 0.5 , 0.2 )
online_weight = 1.0 - offline_weight
for example in batch:
problem = example["problem" ]
image = example["image" ]
target_trace = example["reasoning" ]
generated = self .model.generate(
f"Problem: {problem} " ,
image=image,
max_tokens=2000
)
offline_loss = compute_token_loss(
generated,
target_trace
)
quality = ReflectiveReasoningStructure.score_reasoning_quality(
generated,
extract_answer(problem)
)
reward = quality["quality_score" ]
log_prob = self .model.compute_log_prob(generated)
online_loss = -reward * log_prob
combined_loss = (
offline_weight * offline_loss +
online_weight * online_loss
)
total_loss += combined_loss
entropy = compute_entropy_bonus(
self .model.get_logits(generated)
)
total_loss -= 0.01 * entropy
return total_loss / batch_size, offline_weight, online_weight
def train_ahpo (self, model, dataset, config ):
"""
Execute AHPO training loop.
"""
optimizer = torch.optim.Adam(model.parameters(), lr=1e-5 )
total_steps = config.num_epochs * len (dataset) // config.batch_size
for epoch in range (config.num_epochs):
epoch_loss = 0
for batch_idx in range (0 , len (dataset), config.batch_size):
batch = dataset[batch_idx:batch_idx + config.batch_size]
step = epoch * len (dataset) // config.batch_size + batch_idx // config.batch_size
loss, offline_w, online_w = self .compute_ahpo_loss(
batch,
step,
total_steps
)
optimizer.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0 )
optimizer.step()
epoch_loss += loss.item()
if (batch_idx // config.batch_size) % 100 == 0 :
print (f"Epoch {epoch} , Batch {batch_idx} : "
f"loss={loss:.4 f} , "
f"offline_w={offline_w:.2 f} , online_w={online_w:.2 f} " )
print (f"Epoch {epoch} avg loss: {epoch_loss / len (dataset) * config.batch_size:.4 f} " )
return model
4. Full MM-HELIX Training Pipeline Combine SERG data generation with AHPO training.
def train_mm_helix (base_model, problem_dataset, image_dataset, config ):
"""
Complete MM-HELIX training pipeline.
"""
print ("Step 1: Generating reasoning dataset with SERG" )
serg_generator = StepElicitedResponseGenerator(base_model)
training_dataset = serg_generator.generate_dataset_with_serg(
problem_dataset,
image_dataset,
num_samples=100000
)
quality_filtered = [
ex for ex in training_dataset
if ex["quality_score" ] > 0.5
]
print (f"Kept {len (quality_filtered)} high-quality examples" )
print ("\nStep 2: Training with Adaptive Hybrid Policy Optimization" )
ahpo_trainer = AdaptiveHybridPolicyOptimization(base_model)
trained_model = ahpo_trainer.train_ahpo(
base_model,
quality_filtered,
config
)
return trained_model
5. Evaluation on Reflective Reasoning Benchmarks Assess model performance on iterative reasoning tasks.
def evaluate_mm_helix (model, benchmark_dataset ):
"""
Evaluate model on long-chain reflective reasoning tasks.
"""
results = {
"accuracy" : 0 ,
"avg_iterations" : 0 ,
"avg_reasoning_length" : 0
}
correct = 0
total_iterations = []
reasoning_lengths = []
for example in benchmark_dataset:
problem = example["problem" ]
image = example["image" ]
ground_truth = example["ground_truth" ]
trace = model.generate(
f"Problem: {problem} " ,
image=image,
max_tokens=2000
)
final_answer = extract_final_answer(trace)
if final_answer == ground_truth:
correct += 1
iterations = trace.count("BACKTRACK" )
total_iterations.append(iterations)
reasoning_lengths.append(len (trace.split()))
results["accuracy" ] = correct / len (benchmark_dataset) * 100
results["avg_iterations" ] = np.mean(total_iterations)
results["avg_reasoning_length" ] = np.mean(reasoning_lengths)
print (f"Accuracy: {results['accuracy' ]:.1 f} %" )
print (f"Avg iterations: {results['avg_iterations' ]:.1 f} " )
print (f"Avg reasoning length: {results['avg_reasoning_length' ]:.0 f} tokens" )
return results
Practical Guidance
Offline weight decay : Start at 0.7, decay to 0.2 over trainingOnline weight growth : Start at 0.3, grow to 0.8 over trainingEntropy bonus : 0.01 (prevent mode collapse)Learning rate : 1e-5 (conservative for multimodal models)Batch size : 32-64 (depends on model size)
Training multimodal models for reasoning tasks
Want iterative refinement and backtracking capability
Have access to image+text paired data
Downstream applications require error correction
Single-turn visual question answering
Real-time inference (reflective reasoning adds latency)
Limited compute for large-scale SERG generation
Tasks without multiple solution paths
Reference 