ARIA: Training Language Agents with Intention-Driven Reward Aggregation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910985706536960 |
|---|---|
| author | Yang, Ruihan Zhang, Yikai Chen, Aili Wang, Xintao Yuan, Siyu Chen, Jiangjie Yang, Deqing Xiao, Yanghua |
| author_facet | Yang, Ruihan Zhang, Yikai Chen, Aili Wang, Xintao Yuan, Siyu Chen, Jiangjie Yang, Deqing Xiao, Yanghua |
| contents | Large language models (LLMs) have enabled agents to perform complex reasoning and decision-making through free-form language interactions. However, in open-ended language action environments (e.g., negotiation or question-asking games), the action space can be formulated as a joint distribution over tokens, resulting in an exponentially large action space. Sampling actions in such a space can lead to extreme reward sparsity, which brings large reward variance, hindering effective reinforcement learning (RL). To address this, we propose ARIA, a method that Aggregates Rewards in Intention space to enable efficient and effective language Agents training. ARIA aims to project natural language actions from the high-dimensional joint token distribution space into a low-dimensional intention space, where semantically similar actions are clustered and assigned shared rewards. This intention-aware reward aggregation reduces reward variance by densifying reward signals, fostering better policy optimization. Extensive experiments demonstrate that ARIA not only significantly reduces policy gradient variance, but also delivers substantial performance gains of an average of 9.95% across four downstream tasks, consistently outperforming offline and online RL baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_00539 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ARIA: Training Language Agents with Intention-Driven Reward Aggregation Yang, Ruihan Zhang, Yikai Chen, Aili Wang, Xintao Yuan, Siyu Chen, Jiangjie Yang, Deqing Xiao, Yanghua Computation and Language Large language models (LLMs) have enabled agents to perform complex reasoning and decision-making through free-form language interactions. However, in open-ended language action environments (e.g., negotiation or question-asking games), the action space can be formulated as a joint distribution over tokens, resulting in an exponentially large action space. Sampling actions in such a space can lead to extreme reward sparsity, which brings large reward variance, hindering effective reinforcement learning (RL). To address this, we propose ARIA, a method that Aggregates Rewards in Intention space to enable efficient and effective language Agents training. ARIA aims to project natural language actions from the high-dimensional joint token distribution space into a low-dimensional intention space, where semantically similar actions are clustered and assigned shared rewards. This intention-aware reward aggregation reduces reward variance by densifying reward signals, fostering better policy optimization. Extensive experiments demonstrate that ARIA not only significantly reduces policy gradient variance, but also delivers substantial performance gains of an average of 9.95% across four downstream tasks, consistently outperforming offline and online RL baselines. |
| title | ARIA: Training Language Agents with Intention-Driven Reward Aggregation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.00539 |