treegrpo-tree-advantage-rl

name	treegrpo-tree-advantage-rl
title	TreeGRPO: Tree-Advantage GRPO for Online RL Post-Training of Diffusion Models
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.08153
keywords	["reinforcement learning","diffusion models","tree search","advantage estimation","post-training"]
description	Achieve 2.4× faster RL training for diffusion models by restructuring denoising as tree search with shared computation. TreeGRPO computes fine-grained step advantages instead of trajectory-level rewards—crucial for efficient diffusion model optimization.

Overview

TreeGRPO recasts diffusion model denoising as a tree search problem where multiple candidate trajectories share common prefixes, enabling amortized computation and fine-grained credit assignment through step-specific advantages.

When to Use

• Diffusion model post-training with RL
• Scenarios requiring sample efficiency
• Need for 2.4× speedup in training
• Fine-grained credit assignment important
• Flow-based and diffusion-based models

When NOT to Use

• Models already achieving good results
• Scenarios with limited compute for tree expansion
• Applications where trajectory-level rewards suffice

Core Technique

Tree-structured trajectory generation with advantage backpropagation:

# Tree-structured RL for diffusion models
class TreeGRPO:
    def __init__(self, diffusion_model):
        self.model = diffusion_model
        self.tree = ReasoingTree()

    def build_trajectory_tree(self, initial_noise, reward_fn, num_children=4):
        """Build tree where branches share computation."""
        root = TreeNode(initial_noise)

        # BFS tree expansion
        queue = [root]

        for node in queue:
            if node.depth >= max_depth:
                continue

            # Generate multiple children trajectories
            for child_idx in range(num_children):
                # Each child is denoising step from parent
                child_noise = self.model.denoise(
                    node.noise,
                    step=node.depth
                )

                child = TreeNode(
                    child_noise,
                    parent=node,
                    depth=node.depth + 1
                )

                
                 child.depth == max_depth:
                    child.reward = reward_fn(child.noise)

                node.children.append(child)
                queue.append(child)

         root

     ():
        
        
        .dfs_advantage_computation(tree.root, )

     ():
        
         node.is_leaf:
            node.value = node.reward
        :
            
            child_values = [
                .dfs_advantage_computation(child, )
                 child  node.children
            ]
            node.value = torch.mean(torch.tensor(child_values))

        
         parent_value   :
            node.advantage = node.value - parent_value
        :
            node.advantage = node.value

         node.value