synthrl-visual-reasoning

name	synthrl-visual-reasoning
title	SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesis
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.02096
keywords	["data synthesis","visual reasoning","reinforcement learning","vision-language models","mathematical reasoning"]
description	Scale visual reasoning via automated synthesis of challenging questions from seed samples, using verification mechanisms to ensure correctness and verify RL training gains on out-of-domain visual math tasks.

SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesis

Core Concept

SynthRL addresses data scarcity in visual reasoning tasks by automatically synthesizing challenging training questions from a seed dataset. Rather than manually creating thousands of visual math problems, the framework programmatically generates harder variants and verifies their correctness through automatic checkers.

The approach synthesizes 3,300+ additional challenging questions from approximately 8,000 seed samples, then trains vision-language models using reinforcement learning with verifiable reward signals. The key advantage is that synthesized data gains compound across test-time compute scaling, with improvements most pronounced on the hardest evaluation samples.

Architecture Overview

• Seed Question Selection: Identify representative problems from original dataset
• Augmentation Pipeline: Automatically generate harder variants through question transformation
• Verification Mechanism: Validate correctness of synthesized questions via automated checking
• RLVR Framework: Reinforcement learning with verifiable rewards (not learned rewards)
• Out-of-Domain Testing: Evaluate on benchmarks different from training distribution
• Scalable Data Generation: Generate thousands of new questions with minimal manual effort

Implementation

The following steps outline how to implement verifiable data synthesis for visual reasoning:

1. Prepare seed dataset - Collect initial visual reasoning problems with verified answers
2. Select candidates for augmentation - Identify problems suitable for difficulty increase
3. Generate harder variants - Apply transformations to create more challenging versions
4. Verify correctness - Automatically check that generated questions are correct and harder
5. Prepare RL training - Create question-answer pairs with verifiable reward signals
6. Train with RL - Optimize model using reinforcement learning on synthetic data
7. Evaluate on benchmarks - Test on out-of-domain visual reasoning tasks

from typing import List, Dict, Tuple, Optional
import torch

 :
    

     ():
        .model = base_model
        .templates = transformer_templates

     () -> []:
        
        num_select = ((seed_questions) * selection_ratio)
        
        sorted_by_difficulty = (seed_questions,
                                     key= x: x.get(, ))
        candidates = (sorted_by_difficulty[:num_select//] +
                     sorted_by_difficulty[-num_select//:])
         candidates

     () -> []:
        
        original_text = question[]
        image = question[]
        answer = question[]

        
        augmentations = {
            : ,
            : ,
            : ,
            : 
        }

        augmented_text = augmentations.get(transformation, original_text)

         {
            : augmented_text,
            : image,
            : answer,
            : transformation
        }

     () -> []:
        
        candidates = .select_candidates(seed_data)
        synthetic = []

         question  candidates:
             i, template  (.templates[:augmentations_per_question]):
                aug_question = .augment_question(question, template)
                 aug_question:
                    synthetic.append(aug_question)

         synthetic


 :
    

     ():
        .solver = reference_solver

     () -> [, []]:
        
        :
            predicted_answer = .solver.solve(question[], question[])
            expected_answer = question.get()

            
            is_correct = ._compare_answers(predicted_answer, expected_answer)
            confidence = ._estimate_confidence(predicted_answer)

             is_correct, confidence
         Exception  e:
             , 

     () -> :
        
        
        original_str = original_question[]
        synthetic_str = synthetic_question[]

        
        length_ratio = (synthetic_str.split()) / (original_str.split())
         (, length_ratio)

     () -> :
        
         (predicted, (, ))  (expected, (, )):
             (predicted - expected) < 
         (predicted).strip() == (expected).strip()

     () -> :
        
           answer  

     () -> [[], ]:
        
        valid_questions = []
        stats = {: (synthetic_questions), : , : , : }

        difficulties = []
         question  synthetic_questions:
            is_correct, _ = .verify_correctness(question)
              is_correct:
                stats[] += 
                

            
            original = question.get(, {})
            difficulty_gain = .verify_difficulty(original, question)

             difficulty_gain > :  
                valid_questions.append(question)
                difficulties.append(difficulty_gain)
                stats[] += 

        stats[] = (difficulties) / (difficulties)  difficulties  
         valid_questions, stats


 :
    

     ():
        .model = model
        .verifier = verifier

     () -> :
        
        is_correct, confidence = .verifier.verify_correctness(question)

        
         is_correct:
             confidence  confidence  
        :
               

     () -> :
        
        total_loss = 

         question  batch_questions:
            
            predicted = .model.generate(question[], question[])

            
            reward = .compute_reward(question, predicted)

            
            
            loss = -torch.tensor(reward, dtype=torch.float32)
            total_loss += loss.item()

            
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

         total_loss / (batch_questions)

     () -> :
        
        optimizer = torch.optim.Adam(.model.parameters(), lr=learning_rate)

        metrics = {: [], : , : }

         epoch  (num_epochs):
            epoch_loss = 
             i  (, (synthetic_dataset), batch_size):
                batch = synthetic_dataset[i:i+batch_size]
                loss = .rl_train_step(batch, optimizer)
                epoch_loss += loss

            avg_loss = epoch_loss / ((synthetic_dataset) // batch_size)
            metrics[].append(avg_loss)
            ()

         metrics


 :
    

     ():
        .synthesizer = QuestionSynthesizer(base_model, templates)
        .verifier = QuestionVerifier(reference_solver)
        .trainer = VisualReasoningRLTrainer(base_model, .verifier)

     () -> :
        
        ()
        synthetic = .synthesizer.generate_synthetic_dataset(seed_data)

        ()
        valid_synthetic, stats = .verifier.filter_synthetic_data(synthetic)
        ()

        ()
        metrics = .trainer.train(valid_synthetic, num_epochs=num_epochs)

         {
            : stats,
            : metrics,
            : (valid_synthetic)
        }