Partial Policy Gradients for RL in LLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910043596652544 |
|---|---|
| author | Mathur, Puneet Kveton, Branislav Mukherjee, Subhojyoti Lai, Viet Dac |
| author_facet | Mathur, Puneet Kveton, Branislav Mukherjee, Subhojyoti Lai, Viet Dac |
| contents | Reinforcement learning is a framework for learning to act sequentially in an unknown environment. We propose a natural approach for modeling policy structure in policy gradients. The key idea is to optimize for a subset of future rewards: smaller subsets represent simpler policies, which can be learned more reliably because their empirical gradient estimates are more accurate. Our approach allows for modeling and comparison of different policy classes, including full planning, greedy, K-step lookahead, and segment policies. We evaluate the policies empirically on multiple persona-alignment conversational problems. Different policies excel in different problems, reflecting their different characteristics and highlighting the importance of our studied policy class. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_06138 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Partial Policy Gradients for RL in LLMs Mathur, Puneet Kveton, Branislav Mukherjee, Subhojyoti Lai, Viet Dac Machine Learning Artificial Intelligence Reinforcement learning is a framework for learning to act sequentially in an unknown environment. We propose a natural approach for modeling policy structure in policy gradients. The key idea is to optimize for a subset of future rewards: smaller subsets represent simpler policies, which can be learned more reliably because their empirical gradient estimates are more accurate. Our approach allows for modeling and comparison of different policy classes, including full planning, greedy, K-step lookahead, and segment policies. We evaluate the policies empirically on multiple persona-alignment conversational problems. Different policies excel in different problems, reflecting their different characteristics and highlighting the importance of our studied policy class. |
| title | Partial Policy Gradients for RL in LLMs |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2603.06138 |