Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Dataset
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913707658838016 |
|---|---|
| author | Yang, Yiqin Wang, Quanwei Li, Chenghao Hu, Hao Wu, Chengjie Jiang, Yuhua Zhong, Dianyu Zhang, Ziyou Zhao, Qianchuan Zhang, Chongjie Bo, Xu |
| author_facet | Yang, Yiqin Wang, Quanwei Li, Chenghao Hu, Hao Wu, Chengjie Jiang, Yuhua Zhong, Dianyu Zhang, Ziyou Zhao, Qianchuan Zhang, Chongjie Bo, Xu |
| contents | Offline reinforcement learning (RL) represents a significant shift in RL research, allowing agents to learn from pre-collected datasets without further interaction with the environment. A key, yet underexplored, challenge in offline RL is selecting an optimal subset of the offline dataset that enhances both algorithm performance and training efficiency. Reducing dataset size can also reveal the minimal data requirements necessary for solving similar problems. In response to this challenge, we introduce ReDOR (Reduced Datasets for Offline RL), a method that frames dataset selection as a gradient approximation optimization problem. We demonstrate that the widely used actor-critic framework in RL can be reformulated as a submodular optimization objective, enabling efficient subset selection. To achieve this, we adapt orthogonal matching pursuit (OMP), incorporating several novel modifications tailored for offline RL. Our experimental results show that the data subsets identified by ReDOR not only boost algorithm performance but also do so with significantly lower computational complexity. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_18955 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Dataset Yang, Yiqin Wang, Quanwei Li, Chenghao Hu, Hao Wu, Chengjie Jiang, Yuhua Zhong, Dianyu Zhang, Ziyou Zhao, Qianchuan Zhang, Chongjie Bo, Xu Machine Learning Offline reinforcement learning (RL) represents a significant shift in RL research, allowing agents to learn from pre-collected datasets without further interaction with the environment. A key, yet underexplored, challenge in offline RL is selecting an optimal subset of the offline dataset that enhances both algorithm performance and training efficiency. Reducing dataset size can also reveal the minimal data requirements necessary for solving similar problems. In response to this challenge, we introduce ReDOR (Reduced Datasets for Offline RL), a method that frames dataset selection as a gradient approximation optimization problem. We demonstrate that the widely used actor-critic framework in RL can be reformulated as a submodular optimization objective, enabling efficient subset selection. To achieve this, we adapt orthogonal matching pursuit (OMP), incorporating several novel modifications tailored for offline RL. Our experimental results show that the data subsets identified by ReDOR not only boost algorithm performance but also do so with significantly lower computational complexity. |
| title | Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Dataset |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2502.18955 |