multiagent-process-rewards

name	multiagent-process-rewards
title	Scaling Multiagent Systems with Process Rewards
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2601.23228
keywords	["Multiagent Systems","Process Rewards","Credit Assignment","REINFORCE","Pipeline Agents"]
description	Train specialized agents in pipelines using dense per-action process rewards from AI coaching. Solves credit assignment in sequential workflows, enabling better generalization and faster convergence than outcome-only training.

MAPPA: Process Rewards for Multiagent Pipelines

Problem

Multiagent pipelines suffer from credit assignment problems. When downstream agents encounter errors from upstream work, outcome-based rewards penalize the reporting agent rather than the source of error. This creates incorrect learning signals.

GRPO assumes consistent rollouts from fixed initial states, but multiagent systems produce different intermediate states from the same initial prompt due to upstream stochasticity.

Core Concept

MAPPA (MultiAgent Per-Action Process rewards) trains agents using dense 0-10 scale quality ratings from a coach LLM that evaluates each intermediate action. The coach assesses actions holistically considering role, input, action, and execution results.

This provides correct attribution—upstream errors receive low scores directly, and downstream agents can report issues without penalty.

Architecture Overview

• Coach LLM: Evaluates each agent action independently on 0-10 quality scale
• Per-Action Supervision: Every intermediate step gets explicit quality feedback
• Role-Aware Evaluation: Coach considers agent's role and input context
• Tool Execution Verification: Coach validates whether actions achieved intended effects
• REINFORCE++ Training: Global batch normalization instead of GRPO (accounts for stochastic inputs)

Implementation

Step 1: Define Coach Evaluation Prompt

Structure coach LLM to provide consistent 0-10 quality ratings.

def create_coach_evaluation_prompt(agent_role, input_context, action, execution_result):
    """Create prompt for coach to evaluate action quality."""
    prompt = f"""Evaluate this action on a scale of 0-10.

Agent Role: {agent_role}
Input Context: {input_context}
Action: {action}
Execution Result: {execution_result}

Rate the quality of this action considering:
1. Correctness: Did it achieve intended effect?
2. Efficiency: Did it take optimal approach?
3. Robustness: Did it handle edge cases?

Score (0-10):"""

    return prompt

Step 2: Collect Per-Action Rewards

Call coach for each agent action to obtain dense supervision.

def ():
    
    rewards = []

     step  agent_trajectory:
        
        eval_prompt = create_coach_evaluation_prompt(
            step[],
            step[],
            step[],
            step[]
        )

        evaluation = coach_model.evaluate(eval_prompt)
        score = extract_score(evaluation)  
        rewards.append(score)

     rewards