svs-variational-problem-synthesis

name	svs-variational-problem-synthesis
title	Self-Play with Variational Problem Synthesis for Sustained Reasoning Diversity
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2508.14029
keywords	["reinforcement-learning","problem-synthesis","entropy-preservation","diversity","self-play"]
description	Generate problem variants from correct model solutions while preserving answer equivalence, enabling self-play training that maintains output diversity and prevents entropy collapse.

Self-Play with Variational Problem Synthesis (SvS)

Core Concept

SvS addresses entropy collapse in reasoning RL by synthesizing problem variants derived from correct model solutions. The key insight is that while Pass@1 improves with RL training, Pass@k degrades due to reduced output diversity. By generating new problems from correct solutions (maintaining identical answers), SvS sustains policy entropy during training. This approach achieves 18-22% improvements on AIME benchmarks while preserving the model's ability to generate diverse high-quality solutions.

Architecture Overview

• Correct Solution Extraction: Identify and isolate correct reasoning steps
• Invariant Answer Identification: Determine what must remain constant across variants
• Problem Variant Generation: Create mathematically equivalent problems
• Self-Play Training Loop: Iteratively improve on synthetic problems
• Entropy Preservation: Monitor and maintain output diversity metrics

Implementation Steps

1. Extract and Analyze Correct Solutions

Identify successful reasoning traces for variant generation:

from typing import List, Dict, Tuple, Optional
from dataclasses import dataclass

@dataclass
class Solution:
    """Representation of a model solution."""
    problem: str
    reasoning_steps: List[str]
    final_answer: str
    intermediate_values: Dict[str, any]
    confidence: float
    is_correct: bool

class SolutionAnalyzer:
    def __init__(self, verifier: "AnswerVerifier"):
        .verifier = verifier

     () -> [Solution]:
        
        correct_solutions = []

         problem, response, predicted_answer  samples:
            
            is_correct = .verifier.verify(predicted_answer, problem)

             is_correct:
                
                solution = Solution(
                    problem=problem,
                    reasoning_steps=._parse_steps(response),
                    final_answer=predicted_answer,
                    intermediate_values=._extract_values(response),
                    confidence=  is_correct  ,
                    is_correct=is_correct
                )
                correct_solutions.append(solution)

        
        correct_solutions.sort(key= s: s.confidence, reverse=)
         correct_solutions[:num_best]

     () -> []:
        
        
         re
        steps = re.split(, response)
         [s.strip()  s  steps  s.strip()]

     () -> [, ]:
        
         re
        
        pattern = 
        matches = re.findall(pattern, response)
         {k: (v)  k, v  matches}