grao-unified-alignment

name	grao-unified-alignment
title	Learning to Align Aligning to Learn - GRAO Unified Self-Optimized Alignment
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2508.07750
keywords	["model-alignment","sft-rl","preference-optimization","group-relative-rewards"]
description	Unifies supervised fine-tuning and reinforcement learning through GRAO framework that combines multiple-output generation with group direct alignment loss for improved preference learning.

Learning to Align Aligning to Learn: GRAO Unified Self-Optimized Alignment

Core Concept

GRAO (Group Relative Alignment Optimization) bridges the gap between SFT and RL-based alignment approaches. Rather than treating them as separate phases, GRAO generates multiple candidate outputs, compares them using reward feedback, and applies group-relative optimization to achieve superior alignment with human preferences compared to standard approaches.

Architecture Overview

• Multi-Sample Generation: Create multiple outputs for comparative evaluation
• Group Direct Alignment Loss: Intra-group relative advantage weighting
• Reference-Aware Updates: Pairwise preference dynamics
• Unified Alignment: Single framework combining SFT and RL strengths

Implementation Steps

Step 1: Implement Multi-Sample Generation

Generate candidate responses:

class MultiSampleGenerator:
    def __init__(self, model, num_samples=4):
        super().__init__()
        self.model = model
        self.num_samples = num_samples

    def generate_candidates(self, prompt):
        """Generate multiple response candidates."""
        candidates = []

        for sample_idx in range(self.num_samples):
            with torch.no_grad():
                output = self.model.generate(
                    prompt,
                    max_length=256,
                    temperature=0.8 + 0.1 * sample_idx,  # Varied temperature
                    top_p=0.9,
                    do_sample=True
                )

            text = self.model.tokenizer.decode(output[0])
            candidates.append({
                'text': text,
                'tokens': output[0],
                'index': sample_idx
            })

        return candidates

Step 2: Implement Group Direct Alignment Loss

Design loss for preference learning:

class GroupDirectAlignmentLoss(nn.Module):
    def __init__(self, reward_model):
        super().__init__()
        self.reward_model = reward_model

    def compute_loss(self, candidates, scores):
        """
        Compute GRAO loss with intra-group relative weighting.

        Args:
            candidates: List of candidate outputs
            scores: Reward scores for each candidate

        Returns:
            loss: GRAO loss value
        """
        # Rank candidates by score
        sorted_indices = torch.argsort(torch.tensor(scores), descending=True)

        total_loss = 0

        # Pairwise comparisons within group
        for i in range(len(candidates)):
            for j in range(i + 1, len(candidates)):
                idx_i = sorted_indices[i]
                idx_j = sorted_indices[j]

                # Candidate i should be better than j
                candidate_i = candidates[idx_i]
                candidate_j = candidates[idx_j]

                score_i = scores[idx_i]
                score_j = scores[idx_j]

                # Log probabilities
                logits_i = self._get_logits(candidate_i)
                logits_j = self._get_logits(candidate_j)

                log_prob_i = F.log_softmax(logits_i, dim=-1).mean()
                log_prob_j = F.log_softmax(logits_j, dim=-1).mean()

                # Relative advantage
                relative_advantage = (score_i - score_j) / (abs(score_i - score_j) + 1e-8)

                # Loss: improve i relative to j
                pairwise_loss = -log_prob_i * torch.relu(relative_advantage + )

                total_loss = total_loss + pairwise_loss

         total_loss / ((candidates) * ((candidates) - ) / )

     ():
        
         torch.no_grad():
            outputs = .model(candidate[])
         outputs.logits

Step 3: Compute Preference Scores

Evaluate candidate quality:

class PreferenceScorer:
    def __init__(self, reward_model):
        super().__init__()
        self.reward_model = reward_model

    def score_candidates(self, prompt, candidates):
        """
        Score candidates using reward model.

        Args:
            prompt: Original prompt
            candidates: List of candidate outputs

        Returns:
            scores: Reward scores for each candidate
        """
        scores = []

        for candidate in candidates:
            # Compute reward
            full_text = prompt + candidate['text']

            with torch.no_grad():
                reward = self.reward_model.score(full_text)

            scores.append(reward)

        return torch.tensor(scores)

    def compute_preference_pairs(self, candidates, scores):
        """
        Convert scores to preference pairs.

        Returns:
            pairs: List of (better, worse) candidate pairs
        """
        pairs = []

        sorted_indices = torch.argsort(scores, descending=True)

        for i in range(len(sorted_indices)):
            for j in range(i + 1, len(sorted_indices)):
                better_idx = sorted_indices[i]
                worse_idx = sorted_indices[j]

                pairs.append({
                    'better': candidates[better_idx],
                    'worse': candidates[worse_idx],
                    'score_diff': scores[better_idx] - scores[worse_idx]
                })

         pairs

Step 4: Implement GRAO Training Loop

Full alignment training:

class GRAOTrainer:
    def __init__(self, model, reward_model):
        super().__init__()
        self.model = model
        self.reward_model = reward_model
        self.generator = MultiSampleGenerator(model)
        self.scorer = PreferenceScorer(reward_model)
        self.loss_fn = GroupDirectAlignmentLoss(reward_model)

    def train_grao(self, training_data, num_epochs=3):
        """
        Train model using GRAO framework.

        Args:
            training_data: List of prompts and reference answers
            num_epochs: Training epochs

        Returns:
            training_stats: Statistics from training
        """
        optimizer = AdamW(self.model.parameters(), lr=1e-5)
        scheduler = CosineAnnealingLR(optimizer, T_max=num_epochs * len(training_data))

        for epoch in range(num_epochs):
            total_loss = 0

            for prompt, reference in training_data:
                # Generate multiple candidates
                candidates = self.generator.generate_candidates(prompt)

                # Score candidates
                scores = self.scorer.score_candidates(prompt, candidates)

                # Compute GRAO loss
                loss = self.loss_fn.compute_loss(candidates, scores)

                # Backward pass
                optimizer.zero_grad()
                loss.backward()
                torch.nn.utils.clip_grad_norm_(self.model.parameters(), 1.0)
                optimizer.step()
                scheduler.step()

                total_loss += loss.item()

            ()

         .model

     ():
        
        scores = []

         prompt, reference  test_data:
            candidates = .generator.generate_candidates(prompt)
            candidate_scores = .scorer.score_candidates(prompt, candidates)

            
            best_score = candidate_scores.().item()
            scores.append(best_score)

         {
            : np.mean(scores),
            : np.std(scores),
            : {
                : ,  
                : ,
                : ,
                : 
            }
        }

Practical Guidance

Hyperparameters and Configuration:

• Number of samples: 4-8 candidates
• Learning rate: 1e-5 to 5e-5
• Training epochs: 2-5
• Temperature range: 0.7-1.2 for diversity

When to Use GRAO:

• Model alignment requiring superior human preference matching
• Combining benefits of SFT (stability) and RL (optimization)
• Scenarios with strong reward models
• Systems needing improved generalization

When NOT to Use:

• Weak or noisy reward models
• Scenarios where single-sample SFT sufficient
• Limited computational budget (multi-sample overhead)
• When deterministic behavior critical

Implementation Notes:

• Multi-sample generation provides diversity for learning
• Group-relative loss more stable than pairwise
• Quality of reward model critical for success
• Monitor convergence and sample efficiency

Reference

Paper: Learning to Align Aligning to Learn: Unified Self-Optimized Alignment ArXiv: 2508.07750 Performance: 57.70% improvement over SFT, 17.65% over DPO, 7.95% over PPO, 5.18% over GRPO on alignment tasks