Baba Is AI: Break the Rules to Beat the Benchmark
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915487791710208 |
|---|---|
| author | Cloos, Nathan Jens, Meagan Naim, Michelangelo Kuo, Yen-Ling Cases, Ignacio Barbu, Andrei Cueva, Christopher J. |
| author_facet | Cloos, Nathan Jens, Meagan Naim, Michelangelo Kuo, Yen-Ling Cases, Ignacio Barbu, Andrei Cueva, Christopher J. |
| contents | Humans solve problems by following existing rules and procedures, and also by leaps of creativity to redefine those rules and objectives. To probe these abilities, we developed a new benchmark based on the game Baba Is You where an agent manipulates both objects in the environment and rules, represented by movable tiles with words written on them, to reach a specified goal and win the game. We test three state-of-the-art multi-modal large language models (OpenAI GPT-4o, Google Gemini-1.5-Pro and Gemini-1.5-Flash) and find that they fail dramatically when generalization requires that the rules of the game must be manipulated and combined. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_13729 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Baba Is AI: Break the Rules to Beat the Benchmark Cloos, Nathan Jens, Meagan Naim, Michelangelo Kuo, Yen-Ling Cases, Ignacio Barbu, Andrei Cueva, Christopher J. Computation and Language Humans solve problems by following existing rules and procedures, and also by leaps of creativity to redefine those rules and objectives. To probe these abilities, we developed a new benchmark based on the game Baba Is You where an agent manipulates both objects in the environment and rules, represented by movable tiles with words written on them, to reach a specified goal and win the game. We test three state-of-the-art multi-modal large language models (OpenAI GPT-4o, Google Gemini-1.5-Pro and Gemini-1.5-Flash) and find that they fail dramatically when generalization requires that the rules of the game must be manipulated and combined. |
| title | Baba Is AI: Break the Rules to Beat the Benchmark |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2407.13729 |