An Empirical Study on the Effectiveness of Incorporating Offline RL As Online RL Subroutines
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908681454485504 |
|---|---|
| author | Su, Jianhai Luo, Jinzhu Zhang, Qi |
| author_facet | Su, Jianhai Luo, Jinzhu Zhang, Qi |
| contents | We take the novel perspective of incorporating offline RL algorithms as subroutines of tabula rasa online RL. This is feasible because an online learning agent can repurpose its historical interactions as offline dataset. We formalize this idea into a framework that accommodates several variants of offline RL incorporation such as final policy recommendation and online fine-tuning. We further introduce convenient techniques to improve its effectiveness in enhancing online learning efficiency. Our extensive and systematic empirical analyses show that 1) the effectiveness of the proposed framework depends strongly on the nature of the task, 2) our proposed techniques greatly enhance its effectiveness, and 3) existing online fine-tuning methods are overall ineffective, calling for more research therein. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_00383 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | An Empirical Study on the Effectiveness of Incorporating Offline RL As Online RL Subroutines Su, Jianhai Luo, Jinzhu Zhang, Qi Machine Learning Artificial Intelligence We take the novel perspective of incorporating offline RL algorithms as subroutines of tabula rasa online RL. This is feasible because an online learning agent can repurpose its historical interactions as offline dataset. We formalize this idea into a framework that accommodates several variants of offline RL incorporation such as final policy recommendation and online fine-tuning. We further introduce convenient techniques to improve its effectiveness in enhancing online learning efficiency. Our extensive and systematic empirical analyses show that 1) the effectiveness of the proposed framework depends strongly on the nature of the task, 2) our proposed techniques greatly enhance its effectiveness, and 3) existing online fine-tuning methods are overall ineffective, calling for more research therein. |
| title | An Empirical Study on the Effectiveness of Incorporating Offline RL As Online RL Subroutines |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2512.00383 |