mars-rl-multi-agent-reasoning

name	mars-rl-multi-agent-reasoning
title	MarsRL: Multi-Agent Reasoning System via RL with Agentic Pipeline Parallelism
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2511.11373
keywords	["Multi-Agent RL","Reasoning","Pipeline Parallelism","Credit Assignment","Extended Reasoning"]
description	Train multi-agent reasoning systems with decoupled reward signals and pipeline parallelism—enable specialized Solver/Verifier/Corrector agents to iteratively refine solutions without waiting for full trajectories, handling extended reasoning up to 320K tokens.

Coordinate Multi-Agent Reasoning with Decoupled Rewards and Pipeline Parallelism

Extended reasoning requires multiple agents: a Solver generates candidate solutions, a Verifier checks correctness, a Corrector refines errors. Traditional RL training waits for complete trajectories before updating, creating bottlenecks when trajectories span hundreds of thousands of tokens. MarsRL solves this via two innovations:

1. Decoupled Agent-Specific Rewards: Each agent receives rewards based on its individual performance, not shared trajectory outcomes—avoiding credit assignment noise
2. Pipeline Parallelism: Agents begin training as soon as they complete segments (16k tokens), without waiting for full trajectory completion—reducing latency from hours to minutes

This enables efficient training of specialized agents working in concert, where each agent specializes (Solver on generation, Verifier on discrimination, Corrector on refinement) without reward contamination.

Core Concept

Multi-agent reasoning systems decompose complex problems into specialized subtasks. However, monolithic reward signals (single score for entire trajectory) fail to credit individual agents for their contributions—a Solver's poor generation gets blamed on the Verifier, and vice versa.

MarsRL decouples credit assignment: the Solver is rewarded only for solution correctness, the Verifier for discrimination accuracy, the Corrector for successful refinement. Additionally, rather than accumulating complete trajectories (potentially 320k tokens) before training, MarsRL begins training immediately after each 16k-token segment completes. This "segment rollout" approach combined with pipeline parallelism reduces training latency while maintaining RL signal quality.

Architecture Overview

• Specialized Agents: Solver (generation), Verifier (error detection), Corrector (refinement); each decodes up to 64k tokens
• Segment-Based Decoding: Each agent decodes in 16k-token chunks; upon completion, outputs immediately enter training queue
• Decoupled Reward Functions: Solver rewarded on solution correctness, Verifier on discrimination, Corrector on refinement success
• Grouped Agentic Rollouts: Each problem generates 8 solver outputs; subsequent agents sample 2 outputs each for diverse training signals
• Pipeline Training Queue: Training begins on completed segments while downstream agents continue decoding—overlap eliminates bottlenecks

Implementation Steps

Step 1: Define Agent-Specific Rewards. Assign rewards based on individual agent responsibility, not shared outcome.

 :
     ():
        .reference_answers = reference_answers

     ():
        
         .is_correct(solution, reference_answer):
               
        :
             -  

     ():
        
        ground_truth_correct = .is_correct(solution, .reference_answers[problem])

         verifier_judgment == ground_truth_correct:
               
        :
             -  

     ():
        
        original_correct = .is_correct(original_solution, reference_answer)
        corrected_correct = .is_correct(corrected_solution, reference_answer)

          original_correct  corrected_correct:
               
         corrected_correct:
               
        :
             -  

     ():
        
        
         solution.strip() == reference.strip()