Active Advantage-Aligned Online Reinforcement Learning with Offline Data
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912949522661376 |
|---|---|
| author | Liu, Xuefeng Le, Hung T. C. Chen, Siyu Stevens, Rick Yang, Zhuoran Walter, Matthew R. Chen, Yuxin |
| author_facet | Liu, Xuefeng Le, Hung T. C. Chen, Siyu Stevens, Rick Yang, Zhuoran Walter, Matthew R. Chen, Yuxin |
| contents | Online reinforcement learning (RL) enhances policies through direct interactions with the environment, but faces challenges related to sample efficiency. In contrast, offline RL leverages extensive pre-collected data to learn policies, but often produces suboptimal results due to limited data coverage. Recent efforts integrate offline and online RL in order to harness the advantages of both approaches. However, effectively combining online and offline RL remains challenging due to issues that include catastrophic forgetting, lack of robustness to data quality and limited sample efficiency in data utilization. In an effort to address these challenges, we introduce A3RL, which incorporates a novel confidence aware Active Advantage Aligned (A3) sampling strategy that dynamically prioritizes data aligned with the policy's evolving needs from both online and offline sources, optimizing policy improvement. Moreover, we provide theoretical insights into the effectiveness of our active sampling strategy and conduct diverse empirical experiments and ablation studies, demonstrating that our method outperforms competing online RL techniques that leverage offline data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_07937 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Active Advantage-Aligned Online Reinforcement Learning with Offline Data Liu, Xuefeng Le, Hung T. C. Chen, Siyu Stevens, Rick Yang, Zhuoran Walter, Matthew R. Chen, Yuxin Machine Learning Online reinforcement learning (RL) enhances policies through direct interactions with the environment, but faces challenges related to sample efficiency. In contrast, offline RL leverages extensive pre-collected data to learn policies, but often produces suboptimal results due to limited data coverage. Recent efforts integrate offline and online RL in order to harness the advantages of both approaches. However, effectively combining online and offline RL remains challenging due to issues that include catastrophic forgetting, lack of robustness to data quality and limited sample efficiency in data utilization. In an effort to address these challenges, we introduce A3RL, which incorporates a novel confidence aware Active Advantage Aligned (A3) sampling strategy that dynamically prioritizes data aligned with the policy's evolving needs from both online and offline sources, optimizing policy improvement. Moreover, we provide theoretical insights into the effectiveness of our active sampling strategy and conduct diverse empirical experiments and ablation studies, demonstrating that our method outperforms competing online RL techniques that leverage offline data. |
| title | Active Advantage-Aligned Online Reinforcement Learning with Offline Data |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2502.07937 |