Align and Filter: Improving Performance in Asynchronous On-Policy RL
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911478267772928 |
|---|---|
| author | Honari, Homayoun Castanyer, Roger Creus Przystupa, Michael Noukhovitch, Michael Castro, Pablo Samuel Berseth, Glen |
| author_facet | Honari, Homayoun Castanyer, Roger Creus Przystupa, Michael Noukhovitch, Michael Castro, Pablo Samuel Berseth, Glen |
| contents | Distributed training and increasing the gradient update frequency are practical strategies to accelerate learning and improve performance, but both exacerbate a central challenge: \textit{policy lag}, which is the mismatch between the behavior policy generating data and the learning policy being updated. Policy lag can hinder the scaling of on-policy learning algorithms to larger problems. In this paper, we identify the sources of policy lag caused by distributed learning and high update frequency. We use the findings to propose \textit{total Variation-based Advantage aligned Constrained policy Optimization (\methodacronym)} as a practical approach to mitigate policy lag. We empirically validate our method and show that it offers better robustness to policy lag in classic RL tasks and a modern RL for LLM math reasoning task. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_01365 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Align and Filter: Improving Performance in Asynchronous On-Policy RL Honari, Homayoun Castanyer, Roger Creus Przystupa, Michael Noukhovitch, Michael Castro, Pablo Samuel Berseth, Glen Machine Learning Artificial Intelligence Robotics Systems and Control Distributed training and increasing the gradient update frequency are practical strategies to accelerate learning and improve performance, but both exacerbate a central challenge: \textit{policy lag}, which is the mismatch between the behavior policy generating data and the learning policy being updated. Policy lag can hinder the scaling of on-policy learning algorithms to larger problems. In this paper, we identify the sources of policy lag caused by distributed learning and high update frequency. We use the findings to propose \textit{total Variation-based Advantage aligned Constrained policy Optimization (\methodacronym)} as a practical approach to mitigate policy lag. We empirically validate our method and show that it offers better robustness to policy lag in classic RL tasks and a modern RL for LLM math reasoning task. |
| title | Align and Filter: Improving Performance in Asynchronous On-Policy RL |
| topic | Machine Learning Artificial Intelligence Robotics Systems and Control |
| url | https://arxiv.org/abs/2603.01365 |