agentic-critical-training

name	agentic-critical-training
title	Agentic Critical Training: Teaching Agents to Reason About Action Quality
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.08706
keywords	["Agent Training","Reinforcement Learning","Action Selection","Chain of Thought","GRPO"]
description	Improves LLM agent decision-making by training agents to first critically evaluate actions before generating, using RL on action-pair comparisons. Develops intrinsic reasoning about action quality without requiring reflection supervision.

Agentic Critical Training: Developing Intrinsic Action Quality Reasoning

LLM agents trained on imitation learning only learn what to do, not why certain actions are preferable. They never contrast successful actions against alternatives, leaving them unaware of action quality. Traditional reflection-based approaches attempt to address this but remain fundamentally imitative—they copy pre-generated reflections rather than autonomously reasoning about quality.

Agentic Critical Training (ACT) develops genuine action quality reasoning through a two-stage RL approach: first train the agent to critically compare action pairs via RL, then leverage this learned critical ability for direct action generation. This forces autonomous reasoning without supervised reflection supervision.

Core Concept

Stage 1 - Critical Reasoning Training:

• Present expert action vs model-generated alternative at each decision point
• Train via RL to select the better action
• Model develops understanding of quality through comparison rewards only
• No reflection text supervision—only selection outcome matters

Stage 2 - Action Generation Training:

• Fine-tune on direct action generation
• Model now has learned critical foundation—knows what makes actions good
• Better ability to generate high-quality actions from scratch

Key insight: Selection is simpler than generation but teaches reasoning. By training on comparison first, the model internalizes quality criteria before attempting generation.

Architecture Overview

• Comparison Stage: Present action pairs, train selection via RL
• Reward Design: Accuracy rewards + admissibility bonus + format validation
• Critical Understanding: Model learns what distinguishes good actions
• Generation Fine-Tuning: Apply learned understanding to action generation
• Verification Rewards: Use verifiable outcomes rather than external judges

Implementation Steps

Implement two-stage training: comparison selection followed by generation refinement.

Stage 1: Critical Training via Action Comparison

import torch
import torch.nn as nn
from transformers import AutoModelForCausalLM, AutoTokenizer

class CriticalAgentTrainer:
    

     ():
        .model = AutoModelForCausalLM.from_pretrained(model_name)
        .tokenizer = AutoTokenizer.from_pretrained(model_name)
        .device = device
        .model.to(device)

     ():
        
        prompt = 

         prompt

     ():
        
        input_ids = .tokenizer.encode(prompt, return_tensors=).to(.device)

         torch.no_grad():
            outputs = .model(input_ids, output_hidden_states=)
            logits = outputs.logits[:, -, :]  

        
        token_A = .tokenizer.encode()[-]
        token_B = .tokenizer.encode()[-]

        logit_A = logits[, token_A]
        logit_B = logits[, token_B]

        
        probs = torch.softmax(torch.stack([logit_A, logit_B]), dim=)
        selection =   logit_A > logit_B  

         selection, torch.log(probs[  selection ==   ])

     ():
        
        reward_components = {}

        
         selection == :
            reward_components[] = 
        :
            reward_components[] = -

        
        
        non_selected = model_action  selection ==   expert_action
        is_valid_action = ._verify_action_validity(non_selected, state, task)
        reward_components[] =   is_valid_action  -

        
        
        reward_components[] = 

        
        total_reward = (
             * reward_components[] +
             * reward_components[] +
             * reward_components[]
        )

         total_reward

     ():
        
        
        
         action  (action) > :
             
         


 :
    

     ():
        .model = model
        .optimizer = optimizer
        .trainer = trainer

     ():
        
        batch_size = (batch_data)
        group_size = batch_size

        all_rewards = []
        all_log_probs = []

        
         sample  batch_data:
            prompt = .trainer.prepare_comparison_prompt(
                sample[],
                sample[],
                sample[],
                sample[]
            )

            selection, log_prob = .trainer.get_model_selection(prompt)
            all_log_probs.append(log_prob)

            
            reward = .trainer.compute_comparison_reward(
                sample[],
                sample[],
                sample[],
                selection,
                sample[]
            )
            all_rewards.append(reward)

        all_rewards = torch.tensor(all_rewards, device=.model.device)
        all_log_probs = torch.stack(all_log_probs)

        
        advantages = all_rewards - all_rewards.mean()
        advantages = advantages / (all_rewards.std() + )

        
        loss = -(all_log_probs * advantages.detach()).mean()

        
        .optimizer.zero_grad()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(.model.parameters(), max_norm=)
        .optimizer.step()

         loss.item()