Continual Diffuser (CoD): Mastering Continual Offline Reinforcement Learning with Experience Rehearsal
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917892797235200 |
|---|---|
| author | Hu, Jifeng Shen, Li Huang, Sili Yang, Zhejian Chen, Hechang Sun, Lichao Chang, Yi Tao, Dacheng |
| author_facet | Hu, Jifeng Shen, Li Huang, Sili Yang, Zhejian Chen, Hechang Sun, Lichao Chang, Yi Tao, Dacheng |
| contents | Artificial neural networks, especially recent diffusion-based models, have shown remarkable superiority in gaming, control, and QA systems, where the training tasks' datasets are usually static. However, in real-world applications, such as robotic control of reinforcement learning (RL), the tasks are changing, and new tasks arise in a sequential order. This situation poses the new challenge of plasticity-stability trade-off for training an agent who can adapt to task changes and retain acquired knowledge. In view of this, we propose a rehearsal-based continual diffusion model, called Continual Diffuser (CoD), to endow the diffuser with the capabilities of quick adaptation (plasticity) and lasting retention (stability). Specifically, we first construct an offline benchmark that contains 90 tasks from multiple domains. Then, we train the CoD on each task with sequential modeling and conditional generation for making decisions. Next, we preserve a small portion of previous datasets as the rehearsal buffer and replay it to retain the acquired knowledge. Extensive experiments on a series of tasks show CoD can achieve a promising plasticity-stability trade-off and outperform existing diffusion-based methods and other representative baselines on most tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_02512 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Continual Diffuser (CoD): Mastering Continual Offline Reinforcement Learning with Experience Rehearsal Hu, Jifeng Shen, Li Huang, Sili Yang, Zhejian Chen, Hechang Sun, Lichao Chang, Yi Tao, Dacheng Machine Learning Artificial Intelligence Artificial neural networks, especially recent diffusion-based models, have shown remarkable superiority in gaming, control, and QA systems, where the training tasks' datasets are usually static. However, in real-world applications, such as robotic control of reinforcement learning (RL), the tasks are changing, and new tasks arise in a sequential order. This situation poses the new challenge of plasticity-stability trade-off for training an agent who can adapt to task changes and retain acquired knowledge. In view of this, we propose a rehearsal-based continual diffusion model, called Continual Diffuser (CoD), to endow the diffuser with the capabilities of quick adaptation (plasticity) and lasting retention (stability). Specifically, we first construct an offline benchmark that contains 90 tasks from multiple domains. Then, we train the CoD on each task with sequential modeling and conditional generation for making decisions. Next, we preserve a small portion of previous datasets as the rehearsal buffer and replay it to retain the acquired knowledge. Extensive experiments on a series of tasks show CoD can achieve a promising plasticity-stability trade-off and outperform existing diffusion-based methods and other representative baselines on most tasks. |
| title | Continual Diffuser (CoD): Mastering Continual Offline Reinforcement Learning with Experience Rehearsal |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2409.02512 |