OGPO: Sample Efficient Full-Finetuning of Generative Control Policies
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910190700331008 |
|---|---|
| author | Patil, Sarvesh Nakamoto, Mitsuhiko Agarwal, Manan Saxena, Shashwat Zhang, Jesse Anantharaman, Giri Winston, Cleah Pan, Chaoyi Chen, Douglas Huang, Nai-Chieh Temel, Zeynep Kroemer, Oliver Levine, Sergey Gupta, Abhishek Da, Hongkai Shah, Paarth Simchowitz, Max |
| author_facet | Patil, Sarvesh Nakamoto, Mitsuhiko Agarwal, Manan Saxena, Shashwat Zhang, Jesse Anantharaman, Giri Winston, Cleah Pan, Chaoyi Chen, Douglas Huang, Nai-Chieh Temel, Zeynep Kroemer, Oliver Levine, Sergey Gupta, Abhishek Da, Hongkai Shah, Paarth Simchowitz, Max |
| contents | Generative control policies (GCPs), such as diffusion- and flow-based control policies, have emerged as effective parameterizations for robot learning. This work introduces Off-policy Generative Policy Optimization (OGPO), a sample-efficient algorithm for finetuning GCPs that maintains off-policy critic networks to maximize data reuse and propagate policy gradients through the full generative process of the policy via a modified PPO objective, using critics as the terminal reward. OGPO achieves state-of-the-art performance on manipulation tasks spanning multi-task settings, high-precision insertion, and dexterous control. To our knowledge, it is also the only method that can fine-tune poorly-initialized behavior cloning policies to near full task-success with no expert data in the online replay buffer, and does so with few task-specific hyperparameter tuning. Through extensive empirical investigations, we demonstrate the OGPO drastically outperforms methods alternatives on policy steering and learning residual corrections, and identify the key mechanisms behind its performance. We further introduce practical stabilizers, including success-buffer regularization, conservative advantages, $χ^2$ regularization, and Q-variance reduction, to mitigate critic over-exploitation across state- and pixel-based settings. Beyond proposing OGPO, we conduct a systematic empirical study of GCP finetuning, identifying the stabilizing mechanisms and failure modes that govern successful off-policy full-policy improvement. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_03065 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | OGPO: Sample Efficient Full-Finetuning of Generative Control Policies Patil, Sarvesh Nakamoto, Mitsuhiko Agarwal, Manan Saxena, Shashwat Zhang, Jesse Anantharaman, Giri Winston, Cleah Pan, Chaoyi Chen, Douglas Huang, Nai-Chieh Temel, Zeynep Kroemer, Oliver Levine, Sergey Gupta, Abhishek Da, Hongkai Shah, Paarth Simchowitz, Max Machine Learning Robotics Generative control policies (GCPs), such as diffusion- and flow-based control policies, have emerged as effective parameterizations for robot learning. This work introduces Off-policy Generative Policy Optimization (OGPO), a sample-efficient algorithm for finetuning GCPs that maintains off-policy critic networks to maximize data reuse and propagate policy gradients through the full generative process of the policy via a modified PPO objective, using critics as the terminal reward. OGPO achieves state-of-the-art performance on manipulation tasks spanning multi-task settings, high-precision insertion, and dexterous control. To our knowledge, it is also the only method that can fine-tune poorly-initialized behavior cloning policies to near full task-success with no expert data in the online replay buffer, and does so with few task-specific hyperparameter tuning. Through extensive empirical investigations, we demonstrate the OGPO drastically outperforms methods alternatives on policy steering and learning residual corrections, and identify the key mechanisms behind its performance. We further introduce practical stabilizers, including success-buffer regularization, conservative advantages, $χ^2$ regularization, and Q-variance reduction, to mitigate critic over-exploitation across state- and pixel-based settings. Beyond proposing OGPO, we conduct a systematic empirical study of GCP finetuning, identifying the stabilizing mechanisms and failure modes that govern successful off-policy full-policy improvement. |
| title | OGPO: Sample Efficient Full-Finetuning of Generative Control Policies |
| topic | Machine Learning Robotics |
| url | https://arxiv.org/abs/2605.03065 |