Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2406.06485 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909220458201088 |
|---|---|
| author | Wang, Ruoyao Todd, Graham Xiao, Ziang Yuan, Xingdi Côté, Marc-Alexandre Clark, Peter Jansen, Peter |
| author_facet | Wang, Ruoyao Todd, Graham Xiao, Ziang Yuan, Xingdi Côté, Marc-Alexandre Clark, Peter Jansen, Peter |
| contents | Virtual environments play a key role in benchmarking advances in complex planning and decision-making tasks but are expensive and complicated to build by hand. Can current language models themselves serve as world simulators, correctly predicting how actions change different world states, thus bypassing the need for extensive manual coding? Our goal is to answer this question in the context of text-based simulators. Our approach is to build and use a new benchmark, called ByteSized32-State-Prediction, containing a dataset of text game state transitions and accompanying game tasks. We use this to directly quantify, for the first time, how well LLMs can serve as text-based world simulators. We test GPT-4 on this dataset and find that, despite its impressive performance, it is still an unreliable world simulator without further innovations. This work thus contributes both new insights into current LLM's capabilities and weaknesses, as well as a novel benchmark to track future progress as new models appear. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_06485 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Can Language Models Serve as Text-Based World Simulators? Wang, Ruoyao Todd, Graham Xiao, Ziang Yuan, Xingdi Côté, Marc-Alexandre Clark, Peter Jansen, Peter Computation and Language Artificial Intelligence Virtual environments play a key role in benchmarking advances in complex planning and decision-making tasks but are expensive and complicated to build by hand. Can current language models themselves serve as world simulators, correctly predicting how actions change different world states, thus bypassing the need for extensive manual coding? Our goal is to answer this question in the context of text-based simulators. Our approach is to build and use a new benchmark, called ByteSized32-State-Prediction, containing a dataset of text game state transitions and accompanying game tasks. We use this to directly quantify, for the first time, how well LLMs can serve as text-based world simulators. We test GPT-4 on this dataset and find that, despite its impressive performance, it is still an unreliable world simulator without further innovations. This work thus contributes both new insights into current LLM's capabilities and weaknesses, as well as a novel benchmark to track future progress as new models appear. |
| title | Can Language Models Serve as Text-Based World Simulators? |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2406.06485 |