Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.03508 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910247037173760 |
|---|---|
| author | Zhang, Lunjun Han, Shuo Lyu, Hanrui Stadie, Bradly C |
| author_facet | Zhang, Lunjun Han, Shuo Lyu, Hanrui Stadie, Bradly C |
| contents | We introduce D2AC, a new model-free reinforcement learning (RL) algorithm designed to train expressive diffusion policies online effectively. At its core is a policy improvement objective that avoids the high variance of typical policy gradients and the complexity of backpropagation through time. This stable learning process is critically enabled by our second contribution: a robust distributional critic, which we design through a fusion of distributional RL and clipped double Q-learning. The resulting algorithm is highly effective, achieving state-of-the-art performance on a benchmark of eighteen hard RL tasks, including Humanoid, Dog, and Shadow Hand domains, spanning both dense-reward and goal-conditioned RL scenarios. Beyond standard benchmarks, we also evaluate a biologically motivated predator-prey task to examine the behavioral robustness and generalization capacity of our approach. Code: https://github.com/d2ac-actor-critic/d2ac-public |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_03508 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | D2 Actor Critic: Diffusion Actor Meets Distributional Critic Zhang, Lunjun Han, Shuo Lyu, Hanrui Stadie, Bradly C Machine Learning We introduce D2AC, a new model-free reinforcement learning (RL) algorithm designed to train expressive diffusion policies online effectively. At its core is a policy improvement objective that avoids the high variance of typical policy gradients and the complexity of backpropagation through time. This stable learning process is critically enabled by our second contribution: a robust distributional critic, which we design through a fusion of distributional RL and clipped double Q-learning. The resulting algorithm is highly effective, achieving state-of-the-art performance on a benchmark of eighteen hard RL tasks, including Humanoid, Dog, and Shadow Hand domains, spanning both dense-reward and goal-conditioned RL scenarios. Beyond standard benchmarks, we also evaluate a biologically motivated predator-prey task to examine the behavioral robustness and generalization capacity of our approach. Code: https://github.com/d2ac-actor-critic/d2ac-public |
| title | D2 Actor Critic: Diffusion Actor Meets Distributional Critic |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2510.03508 |