Zero-shot Model-based Reinforcement Learning using Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915151517581312 |
|---|---|
| author | Benechehab, Abdelhakim Hili, Youssef Attia El Odonnat, Ambroise Zekri, Oussama Thomas, Albert Paolo, Giuseppe Filippone, Maurizio Redko, Ievgen Kégl, Balázs |
| author_facet | Benechehab, Abdelhakim Hili, Youssef Attia El Odonnat, Ambroise Zekri, Oussama Thomas, Albert Paolo, Giuseppe Filippone, Maurizio Redko, Ievgen Kégl, Balázs |
| contents | The emerging zero-shot capabilities of Large Language Models (LLMs) have led to their applications in areas extending well beyond natural language processing tasks. In reinforcement learning, while LLMs have been extensively used in text-based environments, their integration with continuous state spaces remains understudied. In this paper, we investigate how pre-trained LLMs can be leveraged to predict in context the dynamics of continuous Markov decision processes. We identify handling multivariate data and incorporating the control signal as key challenges that limit the potential of LLMs' deployment in this setup and propose Disentangled In-Context Learning (DICL) to address them. We present proof-of-concept applications in two reinforcement learning settings: model-based policy evaluation and data-augmented off-policy reinforcement learning, supported by theoretical analysis of the proposed methods. Our experiments further demonstrate that our approach produces well-calibrated uncertainty estimates. We release the code at https://github.com/abenechehab/dicl. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_11711 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Zero-shot Model-based Reinforcement Learning using Large Language Models Benechehab, Abdelhakim Hili, Youssef Attia El Odonnat, Ambroise Zekri, Oussama Thomas, Albert Paolo, Giuseppe Filippone, Maurizio Redko, Ievgen Kégl, Balázs Machine Learning The emerging zero-shot capabilities of Large Language Models (LLMs) have led to their applications in areas extending well beyond natural language processing tasks. In reinforcement learning, while LLMs have been extensively used in text-based environments, their integration with continuous state spaces remains understudied. In this paper, we investigate how pre-trained LLMs can be leveraged to predict in context the dynamics of continuous Markov decision processes. We identify handling multivariate data and incorporating the control signal as key challenges that limit the potential of LLMs' deployment in this setup and propose Disentangled In-Context Learning (DICL) to address them. We present proof-of-concept applications in two reinforcement learning settings: model-based policy evaluation and data-augmented off-policy reinforcement learning, supported by theoretical analysis of the proposed methods. Our experiments further demonstrate that our approach produces well-calibrated uncertainty estimates. We release the code at https://github.com/abenechehab/dicl. |
| title | Zero-shot Model-based Reinforcement Learning using Large Language Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2410.11711 |