World Models with Hints of Large Language Models for Goal Achieving
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913386671898624 |
|---|---|
| author | Liu, Zeyuan Huan, Ziyu Wang, Xiyao Lyu, Jiafei Tao, Jian Li, Xiu Huang, Furong Xu, Huazhe |
| author_facet | Liu, Zeyuan Huan, Ziyu Wang, Xiyao Lyu, Jiafei Tao, Jian Li, Xiu Huang, Furong Xu, Huazhe |
| contents | Reinforcement learning struggles in the face of long-horizon tasks and sparse goals due to the difficulty in manual reward specification. While existing methods address this by adding intrinsic rewards, they may fail to provide meaningful guidance in long-horizon decision-making tasks with large state and action spaces, lacking purposeful exploration. Inspired by human cognition, we propose a new multi-modal model-based RL approach named Dreaming with Large Language Models (DLLM). DLLM integrates the proposed hinting subgoals from the LLMs into the model rollouts to encourage goal discovery and reaching in challenging tasks. By assigning higher intrinsic rewards to samples that align with the hints outlined by the language model during model rollouts, DLLM guides the agent toward meaningful and efficient exploration. Extensive experiments demonstrate that the DLLM outperforms recent methods in various challenging, sparse-reward environments such as HomeGrid, Crafter, and Minecraft by 27.7\%, 21.1\%, and 9.9\%, respectively. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_07381 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | World Models with Hints of Large Language Models for Goal Achieving Liu, Zeyuan Huan, Ziyu Wang, Xiyao Lyu, Jiafei Tao, Jian Li, Xiu Huang, Furong Xu, Huazhe Artificial Intelligence Machine Learning Reinforcement learning struggles in the face of long-horizon tasks and sparse goals due to the difficulty in manual reward specification. While existing methods address this by adding intrinsic rewards, they may fail to provide meaningful guidance in long-horizon decision-making tasks with large state and action spaces, lacking purposeful exploration. Inspired by human cognition, we propose a new multi-modal model-based RL approach named Dreaming with Large Language Models (DLLM). DLLM integrates the proposed hinting subgoals from the LLMs into the model rollouts to encourage goal discovery and reaching in challenging tasks. By assigning higher intrinsic rewards to samples that align with the hints outlined by the language model during model rollouts, DLLM guides the agent toward meaningful and efficient exploration. Extensive experiments demonstrate that the DLLM outperforms recent methods in various challenging, sparse-reward environments such as HomeGrid, Crafter, and Minecraft by 27.7\%, 21.1\%, and 9.9\%, respectively. |
| title | World Models with Hints of Large Language Models for Goal Achieving |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2406.07381 |