Yes, Q-learning Helps Offline In-Context RL
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916045686571008 |
|---|---|
| author | Tarasov, Denis Nikulin, Alexander Zisman, Ilya Klepach, Albina Polubarov, Andrei Lyubaykin, Nikita Derevyagin, Alexander Kiselev, Igor Kurenkov, Vladislav |
| author_facet | Tarasov, Denis Nikulin, Alexander Zisman, Ilya Klepach, Albina Polubarov, Andrei Lyubaykin, Nikita Derevyagin, Alexander Kiselev, Igor Kurenkov, Vladislav |
| contents | Existing offline in-context reinforcement learning (ICRL) methods have predominantly relied on supervised training objectives, which are known to have limitations in offline RL settings. In this study, we explore the integration of RL objectives within an offline ICRL framework. Through experiments on more than 150 GridWorld and MuJoCo environment-derived datasets, we demonstrate that optimizing RL objectives directly improves performance by approximately 30% on average compared to widely adopted Algorithm Distillation (AD), across various dataset coverages, structures, expertise levels, and environmental complexities. Furthermore, in the challenging XLand-MiniGrid environment, RL objectives doubled the performance of AD. Our results also reveal that the addition of conservatism during value learning brings additional improvements in almost all settings tested. Our findings emphasize the importance of aligning ICRL learning objectives with the RL reward-maximization goal, and demonstrate that offline RL is a promising direction for advancing ICRL. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_17666 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Yes, Q-learning Helps Offline In-Context RL Tarasov, Denis Nikulin, Alexander Zisman, Ilya Klepach, Albina Polubarov, Andrei Lyubaykin, Nikita Derevyagin, Alexander Kiselev, Igor Kurenkov, Vladislav Machine Learning Artificial Intelligence Existing offline in-context reinforcement learning (ICRL) methods have predominantly relied on supervised training objectives, which are known to have limitations in offline RL settings. In this study, we explore the integration of RL objectives within an offline ICRL framework. Through experiments on more than 150 GridWorld and MuJoCo environment-derived datasets, we demonstrate that optimizing RL objectives directly improves performance by approximately 30% on average compared to widely adopted Algorithm Distillation (AD), across various dataset coverages, structures, expertise levels, and environmental complexities. Furthermore, in the challenging XLand-MiniGrid environment, RL objectives doubled the performance of AD. Our results also reveal that the addition of conservatism during value learning brings additional improvements in almost all settings tested. Our findings emphasize the importance of aligning ICRL learning objectives with the RL reward-maximization goal, and demonstrate that offline RL is a promising direction for advancing ICRL. |
| title | Yes, Q-learning Helps Offline In-Context RL |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2502.17666 |