Improve mathematical problem-solving in multimodal models by decoupling visual perception from inference reasoning. A two-stage pipeline extracts essential visual information and reasoned properties before passing enriched text to inference models, dramatically improving accuracy on visual math problems.
Improve mathematical problem-solving in multimodal models by decoupling visual perception from inference reasoning. A two-stage pipeline extracts essential visual information and reasoned properties before passing enriched text to inference models, dramatically improving accuracy on visual math problems.
Core Concept
MathFlow addresses a fundamental bottleneck in visual mathematical reasoning: most multimodal large language models (MLLMs) "rely more on reading text than seeing diagrams." The solution separates problem-solving into two independent stages—perception and inference—each optimized for its specific task. A specialized perception model first extracts visual information, then a separate inference model performs mathematical reasoning on enriched text representations.
Architecture Overview
The MathFlow system decomposes visual math problem-solving into distinct stages:
FlowVerse Benchmark Framework: Categorizes problem information into four semantic components—Descriptive Information (scene setup), Essential Information (critical visual elements), Reasoned Property (derived observations), and Only Question (the actual query)—enabling systematic evaluation across six problem versions
Perception Stage (MathFlow-P-7B): A specialized 7B parameter model trained to extract critical visual information from diagrams and convert it to high-quality text representations
Inference Stage: Any inference model (GPT-4V, DeepSeek-r1, Gemini-1.5-pro) uses the enriched textual input for mathematical reasoning without needing to "see" the diagram
Implementation
FlowVerse Benchmark Construction
The FlowVerse benchmark systematically evaluates MLLMs by generating six problem versions that vary information availability. This enables measuring how models leverage different information types and where they struggle.
from dataclasses import dataclass
from typing importList, Dictimport json
@dataclassclassMathProblem:
"""Structure for visual mathematical problems with information decomposition."""
descriptive_info: str# Scene/context description
essential_info: str# Critical visual elements from diagram
reasoned_property: str# Derived observations/properties
question: str# The actual question asked
image_path:
ground_truth:
() -> [, ]:
versions = {
: problem.question,
: (
),
: (
),
: (
),
: (
),
: (
),
}
versions
str
# Reference image path
str
# Correct answer
def
generate_flowverse_versions
problem: MathProblem
Dict
str
str
"""Generate six problem versions to evaluate different information types."""
The perception stage uses multi-task pretraining to extract two critical information types. A 3:1 data ratio between Essential Information (EI) and Reasoned Property (RP) tasks leverages larger EI datasets while focusing RP training on the more challenging extraction task.
import torch
import torch.nn as nn
from transformers import AutoModel, AutoTokenizer
import torch.optim as optim
classMathFlowPerceptionModel(nn.Module):
"""Two-task perception model for visual information extraction."""def__init__(self, base_model="Qwen2-VL-7B"):
super().__init__()
self.vision_encoder = AutoModel.from_pretrained(base_model)
self.tokenizer = AutoTokenizer.from_pretrained(base_model)
# Task-specific headsself.ei_head = nn.Linear(4096, 768) # Essential Info headself.rp_head = nn.Linear(4096, 768) # Reasoned Property headself.ei_decoder = nn.Linear(768, 50257) # Token decoder for EIself.rp_decoder = nn.Linear(768, 50257) # Token decoder for RPdefforward(self, image, task_type="ei"):
"""Extract visual information based on task type."""# Encode image to visual features
visual_features = self.vision_encoder(image)
if task_type == "ei":
# Extract essential information (critical visual elements)
ei_embedding = self.ei_head(visual_features)
ei_tokens = self.ei_decoder(ei_embedding)
return ei_tokens
elif task_type == "rp":
# Extract reasoned property (derived observations)
rp_embedding = self.rp_head(visual_features)
rp_tokens = self.rp_decoder(rp_embedding)
return rp_tokens
deftrain_perception_model(
model, train_loader_ei, train_loader_rp,
ei_ratio=0.75, epochs=10, lr=1e-5):
"""Train MathFlow-P with 3:1 EI:RP data ratio."""
optimizer = optim.AdamW(model.parameters(), lr=lr)
criterion = nn.CrossEntropyLoss()
for epoch inrange(epochs):
total_loss = 0# EI pretraining (75% of batches)for batch_idx, (images, ei_targets) inenumerate(train_loader_ei):
if batch_idx / len(train_loader_ei) > ei_ratio:
break
ei_predictions = model(images, task_type="ei")
loss = criterion(ei_predictions, ei_targets)
optimizer.zero_grad()
loss.backward()
optimizer.step()
total_loss += loss.item()
# RP pretraining (25% of batches)for batch_idx, (images, rp_targets) inenumerate(train_loader_rp):
if batch_idx / len(train_loader_rp) > (1 - ei_ratio):
break
rp_predictions = model(images, task_type="rp")
loss = criterion(rp_predictions, rp_targets)
optimizer.zero_grad()
loss.backward()
optimizer.step()
total_loss += loss.item()
print(f"Epoch {epoch}: Loss = {total_loss:.4f}")
return model
Supervised Fine-Tuning for Accuracy
After multi-task pretraining, fine-tune the perception model on the MathFlow-SFT dataset specifically designed to improve accuracy on real math problems. This stage adapts the model to the target domain and task distributions.
defsupervised_finetune_perception(
pretrained_model, sft_dataset, batch_size=16,
num_epochs=5, lr=5e-6):
"""Fine-tune perception model for accuracy on real problems."""
model = pretrained_model
optimizer = optim.AdamW(model.parameters(), lr=lr)
criterion = nn.CrossEntropyLoss()
dataloader = torch.utils.data.DataLoader(
sft_dataset, batch_size=batch_size, shuffle=True
)
for epoch inrange(num_epochs):
model.train()
epoch_loss = 0for images, ei_texts, rp_texts, questions in dataloader:
# Extract EI and RP from visual input
ei_predictions = model(images, task_type="ei")
rp_predictions = model(images, task_type="rp")
# Tokenize ground truth texts
ei_targets = model.tokenizer(
ei_texts, return_tensors="pt", padding=True
)["input_ids"]
rp_targets = model.tokenizer(
rp_texts, return_tensors="pt", padding=True
)["input_ids"]
# Combined loss
loss_ei = criterion(ei_predictions, ei_targets)
loss_rp = criterion(rp_predictions, rp_targets)
loss = loss_ei + loss_rp
optimizer.zero_grad()
loss.backward()
optimizer.step()
epoch_loss += loss.item()
print(f"FT Epoch {epoch}: Loss = {epoch_loss / len(dataloader):.4f}")
return model
MathFlow Inference Pipeline
During inference, use the trained perception model to extract rich textual descriptions from problem images, then pass the enriched text to any inference model for mathematical reasoning. This decoupling allows leveraging state-of-the-art reasoning models.
defmathflow_inference(
perception_model, inference_model,
problem_image, problem_question, use_cot=True):
"""Apply MathFlow pipeline: perception + inference."""# Stage 1: Extract visual information through perception modelwith torch.no_grad():
ei_tokens = perception_model(problem_image, task_type="ei")
rp_tokens = perception_model(problem_image, task_type="rp")
# Decode to text
essential_info = decode_tokens_to_text(ei_tokens)
reasoned_property = decode_tokens_to_text(rp_tokens)
# Stage 2: Construct enriched problem textif use_cot:
enriched_prompt = f"""
Essential Visual Information:
{essential_info}
Derived Properties:
{reasoned_property}
Problem:
{problem_question}
Solve this step-by-step:
"""else:
enriched_prompt = f"""
{essential_info}{reasoned_property}{problem_question}
"""# Stage 3: Perform inference with state-of-the-art model# (GPT-4V, DeepSeek-r1, Gemini-1.5-pro, etc.)
response = inference_model.generate(enriched_prompt, max_tokens=500)
return response
defevaluate_on_flowverse(
perception_model, inference_model,
flowverse_benchmark, num_samples=2000):
"""Evaluate MathFlow across six problem versions."""
results = {}
for version_name in ["v1", "v2", "v3", "v4", "v5", "v6"]:
correct = 0for problem in flowverse_benchmark[:num_samples]:
# Generate answer with specific version
answer = mathflow_inference(
perception_model, inference_model,
problem["image"], problem[version_name]
)
if check_answer_correctness(answer, problem["ground_truth"]):
correct += 1
accuracy = correct / num_samples
results[version_name] = accuracy
print(f"{version_name}: {accuracy:.1%}")
return results
Practical Guidance
When to use MathFlow:
You have visual math problems (geometry, diagrams, charts) where perception is a bottleneck
You want to leverage advanced reasoning models (GPT-4, Gemini, DeepSeek) without expensive fine-tuning
You have limited training data for your specific math domain
You need interpretable extraction of visual information before reasoning
When NOT to use:
Problems are purely text-based without visual diagrams
Your inference model already has strong visual understanding (some cutting-edge multimodal models)
You need real-time single-pass inference (two-stage pipeline adds latency)
Your training data is extremely small (< 10,000 problem-image pairs)
Hyperparameter tuning:
EI:RP ratio: 3:1 ratio shown optimal; adjust to 2:1 if RP extraction is critical for your domain
Pretraining LR: 1e-5 balances convergence and stability; reduce to 5e-6 if training diverges
Fine-tuning LR: 5e-6 is conservative; can increase to 1e-5 if validation metrics plateau
Batch size: Use 16-32 for perception model (constrained by image resolution); scale down on limited VRAM
Common pitfalls:
Mismatch between EI extraction and question type: Geometry problems need precise spatial info, while physics problems need physical property extraction; customize task definitions per domain
Insufficient SFT data: 130,000 SFT samples were critical; with < 50,000, overfitting is likely
Inference model selection: Pairing weak reasoning models with excellent perception extraction still yields poor results; match reasoning capability to problem difficulty
Token imbalance: Very long essential info extractions can exceed context limits; implement truncation strategies
Data sources: MAVIS and Geo170k for EI pretraining (650,000 image-text pairs), custom MathFlow-RP dataset (130,000 samples), evaluation on MathVerse and custom FlowVerse benchmark
Evaluation metrics: Accuracy on FlowVerse (6-version evaluation), MathVerse testmini baseline comparison, ablation analysis of information contribution
Key findings: MLLMs heavily weight text over visual information; RP (reasoned properties) contribute significantly to performance; enhanced perception is a critical bottleneck in mathematical reasoning
Related work: MAVIS (math-aware visual instruction tuning), Geo170k (geometric reasoning), CoT methods for visual reasoning
