The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913701621137408 |
|---|---|
| author | Schoepp, Sheila Jafaripour, Masoud Cao, Yingyue Yang, Tianpei Abdollahi, Fatemeh Golestan, Shadan Sufiyan, Zahin Zaiane, Osmar R. Taylor, Matthew E. |
| author_facet | Schoepp, Sheila Jafaripour, Masoud Cao, Yingyue Yang, Tianpei Abdollahi, Fatemeh Golestan, Shadan Sufiyan, Zahin Zaiane, Osmar R. Taylor, Matthew E. |
| contents | Reinforcement learning (RL) has shown impressive results in sequential decision-making tasks. Meanwhile, Large Language Models (LLMs) and Vision-Language Models (VLMs) have emerged, exhibiting impressive capabilities in multimodal understanding and reasoning. These advances have led to a surge of research integrating LLMs and VLMs into RL. In this survey, we review representative works in which LLMs and VLMs are used to overcome key challenges in RL, such as lack of prior knowledge, long-horizon planning, and reward design. We present a taxonomy that categorizes these LLM/VLM-assisted RL approaches into three roles: agent, planner, and reward. We conclude by exploring open problems, including grounding, bias mitigation, improved representations, and action advice. By consolidating existing research and identifying future directions, this survey establishes a framework for integrating LLMs and VLMs into RL, advancing approaches that unify natural language and visual understanding with sequential decision-making. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_15214 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learning Schoepp, Sheila Jafaripour, Masoud Cao, Yingyue Yang, Tianpei Abdollahi, Fatemeh Golestan, Shadan Sufiyan, Zahin Zaiane, Osmar R. Taylor, Matthew E. Machine Learning Artificial Intelligence Computation and Language Reinforcement learning (RL) has shown impressive results in sequential decision-making tasks. Meanwhile, Large Language Models (LLMs) and Vision-Language Models (VLMs) have emerged, exhibiting impressive capabilities in multimodal understanding and reasoning. These advances have led to a surge of research integrating LLMs and VLMs into RL. In this survey, we review representative works in which LLMs and VLMs are used to overcome key challenges in RL, such as lack of prior knowledge, long-horizon planning, and reward design. We present a taxonomy that categorizes these LLM/VLM-assisted RL approaches into three roles: agent, planner, and reward. We conclude by exploring open problems, including grounding, bias mitigation, improved representations, and action advice. By consolidating existing research and identifying future directions, this survey establishes a framework for integrating LLMs and VLMs into RL, advancing approaches that unify natural language and visual understanding with sequential decision-making. |
| title | The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learning |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2502.15214 |