Solving Continual Offline RL through Selective Weights Activation on Aligned Spaces
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910658100985856 |
|---|---|
| author | Hu, Jifeng Huang, Sili Shen, Li Yang, Zhejian Hu, Shengchao Tang, Shisong Chen, Hechang Chang, Yi Tao, Dacheng Sun, Lichao |
| author_facet | Hu, Jifeng Huang, Sili Shen, Li Yang, Zhejian Hu, Shengchao Tang, Shisong Chen, Hechang Chang, Yi Tao, Dacheng Sun, Lichao |
| contents | Continual offline reinforcement learning (CORL) has shown impressive ability in diffusion-based lifelong learning systems by modeling the joint distributions of trajectories. However, most research only focuses on limited continual task settings where the tasks have the same observation and action space, which deviates from the realistic demands of training agents in various environments. In view of this, we propose Vector-Quantized Continual Diffuser, named VQ-CD, to break the barrier of different spaces between various tasks. Specifically, our method contains two complementary sections, where the quantization spaces alignment provides a unified basis for the selective weights activation. In the quantized spaces alignment, we leverage vector quantization to align the different state and action spaces of various tasks, facilitating continual training in the same space. Then, we propose to leverage a unified diffusion model attached by the inverse dynamic model to master all tasks by selectively activating different weights according to the task-related sparse masks. Finally, we conduct extensive experiments on 15 continual learning (CL) tasks, including conventional CL task settings (identical state and action spaces) and general CL task settings (various state and action spaces). Compared with 16 baselines, our method reaches the SOTA performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_15698 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Solving Continual Offline RL through Selective Weights Activation on Aligned Spaces Hu, Jifeng Huang, Sili Shen, Li Yang, Zhejian Hu, Shengchao Tang, Shisong Chen, Hechang Chang, Yi Tao, Dacheng Sun, Lichao Machine Learning Continual offline reinforcement learning (CORL) has shown impressive ability in diffusion-based lifelong learning systems by modeling the joint distributions of trajectories. However, most research only focuses on limited continual task settings where the tasks have the same observation and action space, which deviates from the realistic demands of training agents in various environments. In view of this, we propose Vector-Quantized Continual Diffuser, named VQ-CD, to break the barrier of different spaces between various tasks. Specifically, our method contains two complementary sections, where the quantization spaces alignment provides a unified basis for the selective weights activation. In the quantized spaces alignment, we leverage vector quantization to align the different state and action spaces of various tasks, facilitating continual training in the same space. Then, we propose to leverage a unified diffusion model attached by the inverse dynamic model to master all tasks by selectively activating different weights according to the task-related sparse masks. Finally, we conduct extensive experiments on 15 continual learning (CL) tasks, including conventional CL task settings (identical state and action spaces) and general CL task settings (various state and action spaces). Compared with 16 baselines, our method reaches the SOTA performance. |
| title | Solving Continual Offline RL through Selective Weights Activation on Aligned Spaces |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2410.15698 |