Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.00623 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913109338226688 |
|---|---|
| author | Ji, Yanbiao Li, Qiuchang Hu, Yuting Wu, Shaokai Xie, Wenyuan Zhang, Guodong He, Qicheng Ji, Deyi Ding, Yue Lu, Hongtao |
| author_facet | Ji, Yanbiao Li, Qiuchang Hu, Yuting Wu, Shaokai Xie, Wenyuan Zhang, Guodong He, Qicheng Ji, Deyi Ding, Yue Lu, Hongtao |
| contents | This paper introduces EnergyFlow, a framework that unifies generative action modeling with inverse reinforcement learning by parameterizing a scalar energy function whose gradient is the denoising field. We establish that under maximum-entropy optimality, the score function learned via denoising score matching recovers the gradient of the expert's soft Q-function, enabling reward extraction without adversarial training. Formally, we prove that constraining the learned field to be conservative reduces hypothesis complexity and tightens out-of-distribution generalization bounds. We further characterize the identifiability of recovered rewards and bound how score estimation errors propagate to action preferences. Empirically, EnergyFlow achieves state-of-the-art imitation performance on various manipulation tasks while providing an effective reward signal for downstream reinforcement learning that outperforms both adversarial IRL methods and likelihood-based alternatives. These results show that the structural constraints required for valid reward extraction simultaneously serve as beneficial inductive biases for policy generalization. The code is available at https://github.com/sotaagi/EnergyFlow. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_00623 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Recovering Hidden Reward in Diffusion-Based Policies Ji, Yanbiao Li, Qiuchang Hu, Yuting Wu, Shaokai Xie, Wenyuan Zhang, Guodong He, Qicheng Ji, Deyi Ding, Yue Lu, Hongtao Robotics This paper introduces EnergyFlow, a framework that unifies generative action modeling with inverse reinforcement learning by parameterizing a scalar energy function whose gradient is the denoising field. We establish that under maximum-entropy optimality, the score function learned via denoising score matching recovers the gradient of the expert's soft Q-function, enabling reward extraction without adversarial training. Formally, we prove that constraining the learned field to be conservative reduces hypothesis complexity and tightens out-of-distribution generalization bounds. We further characterize the identifiability of recovered rewards and bound how score estimation errors propagate to action preferences. Empirically, EnergyFlow achieves state-of-the-art imitation performance on various manipulation tasks while providing an effective reward signal for downstream reinforcement learning that outperforms both adversarial IRL methods and likelihood-based alternatives. These results show that the structural constraints required for valid reward extraction simultaneously serve as beneficial inductive biases for policy generalization. The code is available at https://github.com/sotaagi/EnergyFlow. |
| title | Recovering Hidden Reward in Diffusion-Based Policies |
| topic | Robotics |
| url | https://arxiv.org/abs/2605.00623 |