spark-process-aware-rewards

name	spark-process-aware-rewards
title	SPARK: Stepwise Process-Aware Rewards for Reference-Free Reinforcement Learning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.03244
keywords	["process rewards","reference-free learning","mathematical reasoning","verification","reward models"]
description	Train process reward models without ground-truth references using synthetic verification data from generators and verifiers. SPARK achieves 67.5 F1 on ProcessBench—ideal when step-level annotations are expensive but verification is available.

Overview

SPARK eliminates the need for ground-truth step-level annotations by training process reward models (PRMs) using synthetic verification data. A generator produces diverse solutions while a verifier evaluates them, providing supervision that outperforms ground-truth-based training when used to train generative PRMs.

When to Use

• Training process reward models without expensive step annotations
• Mathematical reasoning and problem-solving tasks
• Scenarios where verification is cheaper than annotation
• Applications needing to identify erroneous reasoning steps
• RL training requiring step-level reward signals

When NOT to Use

• Tasks with abundant ground-truth step-level annotations
• Domains where verification is as expensive as annotation
• Simple tasks not benefiting from step-level rewards
• Outcome-only verification approaches are sufficient

Core Technique

Three-stage framework combining generation, verification, and PRM training:

# Reference-Free Process Reward Model Training
class SPARKFramework:
    def __init__(self, generator, verifier):
        self.generator = generator
        self.verifier = verifier
        self.prm = None

    def stage1_generate_and_verify(self, problems, num_diverse_solutions=10):
        """
        Stage 1: Generate diverse solutions and verify them.
        Creates synthetic supervision without ground truth.
        """
        synthetic_data = []

        for problem in problems:
            # Generate diverse solution trajectories
            solutions = []
            for _ in range(num_diverse_solutions):
                solution = self.generator.generate(
                    problem,
                    temperature=0.7  
                )
                solutions.append(solution)

            
             solution  solutions:
                parallel_result = .verifier.verify_parallel(
                    problem, solution
                )

                
                sequential_result = .verifier.verify_sequential(
                    problem, solution
                )

                
                verification_data = {
                    : problem,
                    : solution,
                    : solution.steps,
                    : parallel_result,
                    : sequential_result,
                    : sequential_result.step_judgments
                }

                synthetic_data.append(verification_data)

         synthetic_data

     ():
        
        
        .prm = GenerativePRM()

         batch  .create_batches(synthetic_data):
            problems = [d[]  d  batch]
            solutions = [d[]  d  batch]
            step_judgments = [d[]  d  batch]

            
            step_predictions = .prm.predict_step_rewards(
                problems,
                solutions
            )

            
            loss = .compute_pairwise_loss(
                step_predictions,
                step_judgments
            )

            
            .prm.train_step(loss)

         .prm

     ():
        
        rl_agent = RLAgent()

         problem  problems_to_solve:
            
            trajectory = rl_agent.generate_with_rl(problem)

            
            step_rewards = []
             i, step  (trajectory.steps):
                
                reward = .prm.predict_reward(
                    problem,
                    trajectory.steps[:i+]
                )

                
                
                verified = .verify_step_reasoning(
                    problem,
                    trajectory.steps[:i+]
                )

                
                  .check_format_constraints(step):
                    reward =   

                step_rewards.append(reward)

            
            rl_agent.update(trajectory, step_rewards)

         rl_agent

     ():
        
        loss = 
         pred, label  (predictions, labels):
            
            loss += torch.nn.functional.binary_cross_entropy(
                pred, label.()
            )
         loss / (predictions)

     ():
        
        
        logical_valid = .check_logical_soundness(steps_so_far)

        
        mathematically_sound = .check_mathematical_validity(
            problem, steps_so_far
        )

         logical_valid  mathematically_sound