A New Perspective on Transformers in Online Reinforcement Learning for Continuous Control
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914094137737216 |
|---|---|
| author | Kachaev, Nikita Zelezetsky, Daniil Cherepanov, Egor Kovelev, Alexey K. Panov, Aleksandr I. |
| author_facet | Kachaev, Nikita Zelezetsky, Daniil Cherepanov, Egor Kovelev, Alexey K. Panov, Aleksandr I. |
| contents | Despite their effectiveness and popularity in offline or model-based reinforcement learning (RL), transformers remain underexplored in online model-free RL due to their sensitivity to training setups and model design decisions such as how to structure the policy and value networks, share components, or handle temporal information. In this paper, we show that transformers can be strong baselines for continuous control in online model-free RL. We investigate key design questions: how to condition inputs, share components between actor and critic, and slice sequential data for training. Our experiments reveal stable architectural and training strategies enabling competitive performance across fully and partially observable tasks, and in both vector- and image-based settings. These findings offer practical guidance for applying transformers in online RL. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_13367 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A New Perspective on Transformers in Online Reinforcement Learning for Continuous Control Kachaev, Nikita Zelezetsky, Daniil Cherepanov, Egor Kovelev, Alexey K. Panov, Aleksandr I. Machine Learning Artificial Intelligence Robotics Despite their effectiveness and popularity in offline or model-based reinforcement learning (RL), transformers remain underexplored in online model-free RL due to their sensitivity to training setups and model design decisions such as how to structure the policy and value networks, share components, or handle temporal information. In this paper, we show that transformers can be strong baselines for continuous control in online model-free RL. We investigate key design questions: how to condition inputs, share components between actor and critic, and slice sequential data for training. Our experiments reveal stable architectural and training strategies enabling competitive performance across fully and partially observable tasks, and in both vector- and image-based settings. These findings offer practical guidance for applying transformers in online RL. |
| title | A New Perspective on Transformers in Online Reinforcement Learning for Continuous Control |
| topic | Machine Learning Artificial Intelligence Robotics |
| url | https://arxiv.org/abs/2510.13367 |