MacGyver: Are Large Language Models Creative Problem Solvers?
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912240649633792 |
|---|---|
| author | Tian, Yufei Ravichander, Abhilasha Qin, Lianhui Bras, Ronan Le Marjieh, Raja Peng, Nanyun Choi, Yejin Griffiths, Thomas L. Brahman, Faeze |
| author_facet | Tian, Yufei Ravichander, Abhilasha Qin, Lianhui Bras, Ronan Le Marjieh, Raja Peng, Nanyun Choi, Yejin Griffiths, Thomas L. Brahman, Faeze |
| contents | We explore the creative problem-solving capabilities of modern LLMs in a novel constrained setting. To this end, we create MACGYVER, an automatically generated dataset consisting of over 1,600 real-world problems deliberately designed to trigger innovative usage of objects and necessitate out-of-the-box thinking. We then present our collection to both LLMs and humans to compare and contrast their problem-solving abilities. MACGYVER is challenging for both groups, but in unique and complementary ways. For instance, humans excel in tasks they are familiar with but struggle with domain-specific knowledge, leading to a higher variance. In contrast, LLMs, exposed to a variety of specialized knowledge, attempt broader problems but fail by proposing physically-infeasible actions. Finally, we provide a detailed error analysis of LLMs, and demonstrate the potential of enhancing their problem-solving ability with novel prompting techniques such as iterative step-wise reflection and divergent-convergent thinking.
This work (1) introduces a fresh arena for intelligent agents focusing on intricate aspects of physical reasoning, planning, and unconventional thinking, which supplements the existing spectrum of machine intelligence; and (2) provides insight into the constrained problem-solving capabilities of both humans and AI. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_09682 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | MacGyver: Are Large Language Models Creative Problem Solvers? Tian, Yufei Ravichander, Abhilasha Qin, Lianhui Bras, Ronan Le Marjieh, Raja Peng, Nanyun Choi, Yejin Griffiths, Thomas L. Brahman, Faeze Computation and Language Artificial Intelligence We explore the creative problem-solving capabilities of modern LLMs in a novel constrained setting. To this end, we create MACGYVER, an automatically generated dataset consisting of over 1,600 real-world problems deliberately designed to trigger innovative usage of objects and necessitate out-of-the-box thinking. We then present our collection to both LLMs and humans to compare and contrast their problem-solving abilities. MACGYVER is challenging for both groups, but in unique and complementary ways. For instance, humans excel in tasks they are familiar with but struggle with domain-specific knowledge, leading to a higher variance. In contrast, LLMs, exposed to a variety of specialized knowledge, attempt broader problems but fail by proposing physically-infeasible actions. Finally, we provide a detailed error analysis of LLMs, and demonstrate the potential of enhancing their problem-solving ability with novel prompting techniques such as iterative step-wise reflection and divergent-convergent thinking. This work (1) introduces a fresh arena for intelligent agents focusing on intricate aspects of physical reasoning, planning, and unconventional thinking, which supplements the existing spectrum of machine intelligence; and (2) provides insight into the constrained problem-solving capabilities of both humans and AI. |
| title | MacGyver: Are Large Language Models Creative Problem Solvers? |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2311.09682 |