grpo-ma-multi-answer-cot-training

name	grpo-ma-multi-answer-cot-training
title	GRPO-MA: Multi-Answer Generation in GRPO for Stable and Efficient Chain-of-Thought Training
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2509.24494
keywords	["GRPO","chain-of-thought","RL training","variance reduction","multi-answer branching","thought sampling","policy optimization","training stability","language models","reasoning"]
description	Stabilize and accelerate chain-of-thought RL training by sampling multiple answers per generated thought. GRPO-MA reduces gradient noise and improves convergence across math, code, vision, and manipulation tasks while cutting computational cost versus naive thought scaling—critical for training reasoning models without explicit value networks.

Stable Chain-of-Thought Training via Multi-Answer Branching

Outcome

Train reasoning models with stable gradients and superior convergence by generating multiple answers for each internal thought, reducing training variance and achieving better performance per compute dollar than scaling thought samples alone.

Problem Context

Standard Group Relative Policy Optimization (GRPO) trains language models to improve chain-of-thought reasoning by sampling multiple thoughts and ranking their answers. However, estimating the true value of a thought—how good is this reasoning path?—becomes increasingly noisy as you scale thought sampling. Researchers discovered you cannot solve this by simply sampling more thoughts (variance floors out). This leaves a gap: how do you get stable, accurate thought-level advantage estimates for reliable RL updates?

The core problem manifests as gradient noise (spiking) during training, leading to optimization instability, slower convergence, and wasted compute. Teams scaling GRPO have resorted to heuristics—sampling multiple answers per thought—but without theoretical understanding of why this works or when it's necessary.

Core Concept

GRPO-MA reveals that tree-style branching (K thoughts × M answers per thought) is not merely helpful but fundamentally necessary. Using multivariate delta method analysis, the authors prove:

• Thought scaling alone (increase K) converges to a hard variance floor; irreducible noise persists
• Answer branching (increase M) monotonically decreases variance toward zero at rate O(1/M)

By aggregating rewards across multiple answers generated from the same thought, you recover a cleaner signal about that thought's true value. The advantage computation then normalizes both thought-level and answer-level signals, producing stable policy gradients.

Architecture Overview

The GRPO-MA pipeline decomposes training into structured steps:

• Thought generation: Sample K independent reasoning paths from the policy, each conditioned on the problem
• Answer branching: For each thought, generate M distinct answers, all starting from that fixed reasoning path
• Reward aggregation: Compute the value of each thought by averaging rewards across its M answers
• Advantage normalization: Compute standardized advantages at both thought level (across K values) and answer level (across K×M total answers)
• Loss computation: Combine thought and answer advantages in a weighted objective, applying PPO clipping and KL divergence penalty

The key insight: this hierarchy—thoughts nested under thoughts, answers under each thought—naturally filters out high-frequency noise while preserving signal about solution quality.