Posterior Optimization with Clipped Objective for Bridging Efficiency and Stability in Generative Policy Learning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911562920361984 |
|---|---|
| author | Chen, Yuhui Li, Haoran Jiang, Zhennan Qin, Yuxing Wan, Yuxuan Liu, Weiheng Zhao, Dongbin |
| author_facet | Chen, Yuhui Li, Haoran Jiang, Zhennan Qin, Yuxing Wan, Yuxuan Liu, Weiheng Zhao, Dongbin |
| contents | Expressive generative models have advanced robotic manipulation by capturing complex, multi-modal action distributions over temporally extended trajectories. However, fine-tuning these policies via RL remains challenging due to instability and sample inefficiency. We introduce Posterior Optimization with Clipped Objective (POCO), a principled RL framework that formulates policy improvement as a posterior inference problem tailored for temporal action chunks. Through an Expectation-Maximization procedure, POCO distills a reward-weighted implicit posterior into the policy without likelihood estimation. Furthermore, POCO adopts an offline-to-online paradigm that anchors online exploration to pre-trained priors, and its model-agnostic design scales to fine-tune large VLA models without architectural modifications. Evaluations across 7 simulation benchmarks and 4 contact-rich real-world tasks demonstrate that POCO prevents catastrophic policy collapse, outperforms SOTA baselines, and achieves a 96.7% success rate on real-world tasks. Videos are available at our project website https://cccedric.github.io/poco/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_01860 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Posterior Optimization with Clipped Objective for Bridging Efficiency and Stability in Generative Policy Learning Chen, Yuhui Li, Haoran Jiang, Zhennan Qin, Yuxing Wan, Yuxuan Liu, Weiheng Zhao, Dongbin Robotics Expressive generative models have advanced robotic manipulation by capturing complex, multi-modal action distributions over temporally extended trajectories. However, fine-tuning these policies via RL remains challenging due to instability and sample inefficiency. We introduce Posterior Optimization with Clipped Objective (POCO), a principled RL framework that formulates policy improvement as a posterior inference problem tailored for temporal action chunks. Through an Expectation-Maximization procedure, POCO distills a reward-weighted implicit posterior into the policy without likelihood estimation. Furthermore, POCO adopts an offline-to-online paradigm that anchors online exploration to pre-trained priors, and its model-agnostic design scales to fine-tune large VLA models without architectural modifications. Evaluations across 7 simulation benchmarks and 4 contact-rich real-world tasks demonstrate that POCO prevents catastrophic policy collapse, outperforms SOTA baselines, and achieves a 96.7% success rate on real-world tasks. Videos are available at our project website https://cccedric.github.io/poco/. |
| title | Posterior Optimization with Clipped Objective for Bridging Efficiency and Stability in Generative Policy Learning |
| topic | Robotics |
| url | https://arxiv.org/abs/2604.01860 |