aria-intention-reward

name	aria-intention-reward
title	ARIA: Training Language Agents with Intention-Driven Reward Aggregation
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.00539
keywords	["reinforcement learning","language agents","reward aggregation","variance reduction","semantic space"]
description	Reduce policy gradient variance in language agent training by aggregating rewards in semantic intention space, enabling 9.95% average performance gains across downstream tasks without exponential action space explosion.

ARIA: Training Language Agents with Intention-Driven Reward Aggregation

Core Concept

ARIA addresses the fundamental challenge of training language agents in open-ended environments where the action space grows exponentially. In traditional reinforcement learning for language agents, each unique token sequence represents a distinct action, creating extremely sparse reward signals that make gradient-based optimization inefficient.

ARIA's key insight is to project natural language actions into a lower-dimensional semantic space where similar actions are grouped together and share reward signals. This intentional aggregation densifies rewards, dramatically reducing policy gradient variance and enabling effective agent training with standard optimization methods.

Architecture Overview

• Intention Space Projection: Convert discrete token distributions from the policy into semantic clusters via dimensionality reduction
• Reward Signal Aggregation: Assign shared rewards to semantically similar actions rather than treating each token sequence independently
• Policy Gradient Optimization: Leverage densified reward signals to improve gradient estimation and reduce variance
• End-to-End Training: Integrate intention space projection as a differentiable layer in the RL pipeline
• Task-Agnostic Design: Apply the same aggregation mechanism across diverse downstream tasks

Implementation

The following steps outline how to implement intention-driven reward aggregation in a language agent training pipeline:

1. Define the intention space encoder - Use a semantic encoder (e.g., a frozen language model or contrastive encoder) to map action descriptions to fixed-size vectors
2. Aggregate reward signals - Group actions by semantic similarity in the intention space and assign shared rewards to clusters
3. Compute policy gradients - Calculate policy gradients using densified rewards to reduce variance
4. Update agent policy - Optimize the language agent using standard PPO or policy gradient methods with aggregated rewards
5. Monitor performance - Track downstream task metrics to validate improvement from reward densification

import torch
import torch.nn as nn
from transformers import AutoModel

class (nn.Module):
     ():
        ().__init__()
        .encoder = AutoModel.from_pretrained(encoder_name)
        .embedding_dim = .encoder.config.hidden_size

     () -> torch.Tensor:
        
        embeddings = .encoder.encode(action_texts, convert_to_tensor=)
         embeddings

     () -> torch.Tensor:
        
        similarity_matrix = torch.nn.functional.cosine_similarity(
            embeddings.unsqueeze(), embeddings.unsqueeze(), dim=
        )
        clusters = (similarity_matrix > clustering_threshold).long()

        aggregated_rewards = torch.zeros_like(rewards)
         i  ((rewards)):
            similar_indices = (clusters[i] == ).nonzero(as_tuple=)[]
            aggregated_rewards[i] = rewards[similar_indices].mean()

         aggregated_rewards

     () -> torch.Tensor:
        
        embeddings = .encode_actions(action_texts)
         .aggregate_rewards(embeddings, rewards)