mint-cot-visual-reasoning

name	mint-cot-visual-reasoning
title	MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.05331
keywords	["multimodal-reasoning","chain-of-thought","visual-language","mathematics","fine-grained-vision"]
description	Integrates fine-grained visual tokens into mathematical reasoning via Interleave Tokens that dynamically select relevant image regions for each reasoning step.

MINT-CoT: Interleaved Visual Tokens in Mathematical Reasoning

Core Concept

Mathematical reasoning with diagrams requires precise alignment between textual reasoning steps and visual regions. Existing approaches use coarse bounding boxes, limiting the fine-grained visual understanding essential for geometry and diagram interpretation. MINT-CoT introduces Interleave Tokens that compute similarity scores between decoder states and visual tokens, enabling dynamic selection of non-rectangular image regions during reasoning. A 54K dataset with token-level alignment and three-stage progressive training enables substantial improvements on mathematical benchmarks.

Architecture Overview

• Interleave Tokens: Special tokens that select relevant visual regions by computing similarity between decoder hidden states and visual token embeddings
• Fine-Grained Selection: Enables non-rectangular region selection, capturing diagram elements at arbitrary shapes
• MINT-CoT Dataset: 54K annotated problems with token-level alignment between reasoning steps and image regions
• Three-Stage Training: Text-only CoT → Interleaved CoT supervised → Interleaved CoT reinforcement learning
• Automated Annotation: Four-step pipeline (gridding, OCR, keyword extraction, alignment) for efficient dataset construction
• GRPO Integration: Reinforcement learning phase optimizes reasoning quality end-to-end

Implementation

The following code demonstrates the Interleave Token mechanism and training pipeline:

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import List, Tuple, Optional

class InterleaveToken(nn.Module):
    """
    Special token that selects relevant visual regions during reasoning.
    """
    def __init__(self, hidden_dim: int, num_visual_tokens: int):
        super().__init__()
        self.hidden_dim = hidden_dim
        self.num_visual_tokens = num_visual_tokens

        # Projections for similarity computation
        self.query_proj = nn.Linear(hidden_dim, hidden_dim)
        self.visual_key_proj = nn.Linear(hidden_dim, hidden_dim)

    def forward(self, decoder_state: torch.Tensor,
               visual_tokens: torch.Tensor,
               threshold: float = 0.5) -> Tuple[torch.Tensor, torch.Tensor]:
        """
        Select visual regions by computing similarity with decoder state.

        decoder_state: (hidden_dim,) hidden state before this reasoning step
        visual_tokens: (num_visual_tokens, hidden_dim) image patches encoded
        threshold: similarity threshold for region selection

        Returns: (selected_regions, selection_mask)
        """
        # Project for similarity computation
        query = self.query_proj(decoder_state)  # (hidden_dim,)
        keys = self.visual_key_proj(visual_tokens)  # (num_visual_tokens, hidden_dim)

        # Compute cosine similarity
        query_norm = F.normalize(query, p=, dim=-)
        keys_norm = F.normalize(keys, p=, dim=-)

        similarity = torch.matmul(keys_norm, query_norm)  

        
        selection_weights = F.softmax(similarity * , dim=)  

        
        selection_mask = (similarity > threshold).()

        
        selected_regions = torch.matmul(
            selection_weights.unsqueeze(), visual_tokens
        )  

         selected_regions, selection_mask


 (nn.Module):
    
     ():
        ().__init__()
        .hidden_dim = hidden_dim
        .vocab_size = vocab_size

        
        .embedding = nn.Embedding(vocab_size, hidden_dim)
        .transformer = nn.TransformerDecoder(
            nn.TransformerDecoderLayer(d_model=hidden_dim, nhead=, dim_feedforward=*hidden_dim,
                                       batch_first=),
            num_layers=
        )
        .output_proj = nn.Linear(hidden_dim, vocab_size)

        
        .vision_encoder = nn.Identity()  
        .interleave_token = InterleaveToken(hidden_dim, num_visual_tokens)

     () -> torch.Tensor:
        
        
        
        visual_tokens = .vision_encoder(image)  
         visual_tokens

     () -> torch.Tensor:
        
        batch_size, seq_len = input_ids.shape

        
        visual_tokens = .encode_image_to_tokens(image)  

        
        text_embeds = .embedding(input_ids)  

        
        interleaved_embeds = text_embeds.clone()

         pos  (seq_len):
             visual_token_mask[:, pos].():
                
                decoder_state = text_embeds[:, pos]  

                
                 b  (batch_size):
                     visual_token_mask[b, pos]:
                        selected_regions, _ = .interleave_token(
                            decoder_state[b], visual_tokens[b]
                        )
                        
                        interleaved_embeds[b, pos] =  * text_embeds[b, pos] +  * selected_regions.squeeze()

        
        output = .transformer(interleaved_embeds, memory=)
        logits = .output_proj(output)

         logits


 :
    
     ():
        .grid_size =   

     () -> [[, , , ]]:
        
        _, h, w = image.shape
        cell_h = h // .grid_size
        cell_w = w // .grid_size

        grid_cells = []
         i  (.grid_size):
             j  (.grid_size):
                x1, y1 = j * cell_w, i * cell_h
                x2, y2 = x1 + cell_w, y1 + cell_h
                grid_cells.append((x1, y1, x2, y2))

         grid_cells

     () -> []:
        
        
        keywords = reasoning_step.split()[:]  
         keywords

     () -> []:
        
        keywords = .extract_keywords_from_step(reasoning_step)
        grid_cells = .grid_image(image)

        
        
        relevant_cells = [, , , ]  

         relevant_cells

     () -> :
        
        
        full_text = problem +  + .join(reasoning_chain) +  + answer
        tokens = full_text.split()  

        
        visual_token_mask = [] * (tokens)

         step_idx, step  (reasoning_chain):
            step_keywords = .extract_keywords_from_step(step)
            
            step_start = ((r.split())  r  reasoning_chain[:step_idx])
            step_end = step_start + (step.split())

            
             pos  (step_start, (step_end, (visual_token_mask))):
                visual_token_mask[pos] = 

         {
            : image,
            : tokens,
            : visual_token_mask,
            : answer
        }


 :
    
     ():
        .model = model
        .optimizer = torch.optim.AdamW(model.parameters(), lr=)

     ():
        
         epoch  (epochs):
             sample  dataset:
                
                logits = .model(torch.tensor(sample[]),
                                  sample[],
                                  torch.zeros_like(sample[]))
                loss = F.cross_entropy(logits.view(-, .model.vocab_size),
                                      torch.tensor(sample[]))
                loss.backward()
                .optimizer.step()
                .optimizer.zero_grad()

     ():
        
         epoch  (epochs):
             sample  dataset:
                
                visual_mask = torch.tensor(sample[]).unsqueeze()
                logits = .model(torch.tensor(sample[]).unsqueeze(),
                                  sample[].unsqueeze(),
                                  visual_mask)

                loss = F.cross_entropy(logits.view(-, .model.vocab_size),
                                      torch.tensor(sample[]))
                loss.backward()
                .optimizer.step()
                .optimizer.zero_grad()

     ():
        
         _  (num_iterations):
            
            
              

Practical Guidance

Grid Size Selection: 16×16 grid (256 cells) provides good granularity for diagram understanding. Increase to 32×32 for high-resolution images with fine details.

Similarity Threshold: Set to 0.5 for 50% similarity threshold in Interleave Token selection. Lower thresholds (0.3) increase region selection; higher (0.7) make selection more selective.

Keyword Extraction: Use GPT-4o for accurate keyword extraction from reasoning steps. Alternatively, use domain-specific keyword lists for specific math domains.

Annotation Quality: The dataset quality is critical. Ensure OCR accuracy and proper alignment between reasoning steps and image regions before training.

Training Schedule: Follow the three-stage progression strictly. Each stage builds on previous; skipping stages hurts performance.

Visualization: Visualize selected image regions during training to verify Interleave Token selection aligns with reasoning steps.

Reference

MINT-CoT achieves substantial improvements on mathematical reasoning:

• MathVista (mathematical subset): +32.59% improvement
• GeoQA: +26.92% improvement
• Geometry-related tasks: Surpasses state-of-the-art models

The 54K annotated dataset with fine-grained visual alignment enables models to leverage diagram information effectively. The approach is particularly valuable for geometry, diagram-based reasoning, and scientific problem-solving where visual information is essential.