Gradient Coupling: The Hidden Barrier to Generalization in Agentic Reinforcement Learning
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917204690206720 |
|---|---|
| author | Liu, Jingyu Wu, Xiaopeng Peng, Jingquan Chen, Kehan Yu, Chuan Ding, Lizhong Liu, Yong |
| author_facet | Liu, Jingyu Wu, Xiaopeng Peng, Jingquan Chen, Kehan Yu, Chuan Ding, Lizhong Liu, Yong |
| contents | Reinforcement learning (RL) is a dominant paradigm for training autonomous agents, yet these agents often exhibit poor generalization, failing to adapt to scenarios not seen during training. In this work, we identify a fundamental cause of this brittleness, a phenomenon which we term "gradient coupling." We hypothesize that in complex agentic tasks, the high similarity between distinct states leads to destructive interference between gradients. Specifically, a gradient update that reinforces an optimal action in one state can inadvertently increase the likelihood of a suboptimal action in a similar, yet different, state. To solve this, we propose a novel objective where the actor is trained to simultaneously function as a classifier that separates good and bad actions. This auxiliary pressure compels the model to learn disentangled embeddings for positive and negative actions, which mitigates negative gradient interference and improve the generalization performance. Extensive experiments demonstrate the effectiveness of our method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_23870 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Gradient Coupling: The Hidden Barrier to Generalization in Agentic Reinforcement Learning Liu, Jingyu Wu, Xiaopeng Peng, Jingquan Chen, Kehan Yu, Chuan Ding, Lizhong Liu, Yong Artificial Intelligence Reinforcement learning (RL) is a dominant paradigm for training autonomous agents, yet these agents often exhibit poor generalization, failing to adapt to scenarios not seen during training. In this work, we identify a fundamental cause of this brittleness, a phenomenon which we term "gradient coupling." We hypothesize that in complex agentic tasks, the high similarity between distinct states leads to destructive interference between gradients. Specifically, a gradient update that reinforces an optimal action in one state can inadvertently increase the likelihood of a suboptimal action in a similar, yet different, state. To solve this, we propose a novel objective where the actor is trained to simultaneously function as a classifier that separates good and bad actions. This auxiliary pressure compels the model to learn disentangled embeddings for positive and negative actions, which mitigates negative gradient interference and improve the generalization performance. Extensive experiments demonstrate the effectiveness of our method. |
| title | Gradient Coupling: The Hidden Barrier to Generalization in Agentic Reinforcement Learning |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2509.23870 |