unigrpo-unified-visual-generation-rl

name	unigrpo-unified-visual-generation-rl
title	UniGRPO: Unified Policy Optimization for Interleaved Text-Image Generation
version	0.0.3
engine	skillxiv-v0.0.3-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.23500
keywords	["Flow Matching","GRPO","Velocity Regularization","CFG Elimination","Multimodal RL"]
description	Replace classifier-free guidance and KL-based regularization in flow matching with velocity-based MSE regularization and GRPO for joint text-image optimization. Achieves 0.8381 TextAlign and 0.90 GenEval without CFG overhead. Works best for multimodal generation where text and image must stay coherent. Trigger: When optimizing vision-language models with flow matching and need better joint text-image policy.
category	Component Innovation

What This Skill Does

Replace two components in multimodal flow-matching models: (1) swap classifier-free guidance (CFG) with direct GRPO optimization during training, and (2) replace KL regularization on latent spaces with MSE regularization on velocity fields. Enables joint text-image policy learning without training-inference mismatch.

Problems with Prior Approach

Problem 1: Classifier-Free Guidance Creates Training-Inference Gap

• CFG requires branching computation during inference (unconditioned + conditioned samples)
• During training with CFG, multiple rollouts per prompt increase computational cost
• Gradient graph complexity from branched rollouts complicates GRPO scaling to multi-turn sequences

Problem 2: KL Regularization on Latents is Crude

• Standard KL penalty applies uniform constraints across all noise levels
• High-noise steps (where model has more freedom) and low-noise steps (where precision matters) get same penalty
• Enables reward hacking: model exploits high-noise regions where KL is loose

The paper's insight: Remove CFG entirely during training (use direct GRPO instead), and apply noise-aware regularization directly on velocity fields rather than latent distributions.

The Swap: Two-Part Component Replacement

Swap 1: CFG → Direct GRPO

# Prior approach: Classifier-free guidance during training
def train_with_cfg(prompts, model, num_rollouts=4):
    """
    CFG requires branched rollouts:
    - Unconditional sample (for guidance scale)
    - Conditional sample (actual generation)
    Doubles forward passes; complicates gradient estimation
    """
    unconditional_outputs = []
    conditional_outputs = []
    for prompt in prompts:
        # Unconditional branch
        unc_sample = model.sample(prompt=None)
        unconditional_outputs.append(unc_sample)

        # Conditional branch
        cond_sample = model.sample(prompt=prompt)
        conditional_outputs.append(cond_sample)

    # CFG interpolation (post-hoc, inference-only)
    # During training: gradient computation through two branches complicates GRPO
    return mix_guidance(conditional_outputs, unconditional_outputs, scale=)


 ():
    
    samples = []
     prompt  prompts:
        sample = model.sample(prompt=prompt)  
        samples.append(sample)

    
    
    advantages = compute_group_relative_advantages(samples)
    loss = -advantages.mean()
     loss