turn-ppo

name	turn-ppo
title	Turn-PPO: Turn-Level Advantage Estimation for Agentic LLMs
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.17008
keywords	["reinforcement-learning","ppo","agents","advantage-estimation","multi-turn"]
description	Stabilize multi-turn agent RL by shifting from token-level to turn-level MDPs. Reformulates states and actions at conversation-turn granularity, uses learned turn-level critics, and applies Generalized Advantage Estimation—eliminating misalignment that destabilizes GRPO training on long-horizon agentic tasks.

Overview

Turn-PPO addresses a fundamental instability in reinforcement learning for multi-turn LLM agents: existing token-level formulations create MDP misalignment with the actual multi-turn interaction structure. By reformulating state-action pairs at turn boundaries, this technique enables stable value estimation and improved credit assignment.

Core Technique

The key insight is that multi-turn interactions have natural episodic structure that token-level approaches ignore.

State-Action Reformulation at Turn Level: Instead of treating individual tokens as actions, entire LLM responses within a turn become single actions, with full conversation history as state.

# Turn-level MDP formulation
class TurnMDP:
    def __init__(self):
        self.turn_states = []  # Full history at each turn
        self.turn_actions = []  # Complete responses per turn

    def add_turn(self, query, response):
        # State: complete history up to current query
        state = self.conversation_history + [query]
        # Action: entire response (all tokens at once)
        action = response
        self.turn_states.append(state)
        self.turn_actions.append(action)

    def compute_advantages(self):
        # Critic operates at turn level, not token level
        values = [self.critic(s) for s in self.turn_states]
        advantages = compute_gae(rewards, values, gamma=0.99)
        return advantages

Learned Turn-Level Critic: Unlike GRPO's sample-based advantage estimation, a separate critic network predicts value for each turn's state, enabling principled GAE computation.