Loose LIPS Sink Ships: Asking Questions in Battleship with Language-Informed Program Sampling
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916231630553088 |
|---|---|
| author | Grand, Gabriel Pepe, Valerio Andreas, Jacob Tenenbaum, Joshua B. |
| author_facet | Grand, Gabriel Pepe, Valerio Andreas, Jacob Tenenbaum, Joshua B. |
| contents | Questions combine our mastery of language with our remarkable facility for reasoning about uncertainty. How do people navigate vast hypothesis spaces to pose informative questions given limited cognitive resources? We study these tradeoffs in a classic grounded question-asking task based on the board game Battleship. Our language-informed program sampling (LIPS) model uses large language models (LLMs) to generate natural language questions, translate them into symbolic programs, and evaluate their expected information gain. We find that with a surprisingly modest resource budget, this simple Monte Carlo optimization strategy yields informative questions that mirror human performance across varied Battleship board scenarios. In contrast, LLM-only baselines struggle to ground questions in the board state; notably, GPT-4V provides no improvement over non-visual baselines. Our results illustrate how Bayesian models of question-asking can leverage the statistics of language to capture human priors, while highlighting some shortcomings of pure LLMs as grounded reasoners. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_19471 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Loose LIPS Sink Ships: Asking Questions in Battleship with Language-Informed Program Sampling Grand, Gabriel Pepe, Valerio Andreas, Jacob Tenenbaum, Joshua B. Computation and Language Artificial Intelligence Questions combine our mastery of language with our remarkable facility for reasoning about uncertainty. How do people navigate vast hypothesis spaces to pose informative questions given limited cognitive resources? We study these tradeoffs in a classic grounded question-asking task based on the board game Battleship. Our language-informed program sampling (LIPS) model uses large language models (LLMs) to generate natural language questions, translate them into symbolic programs, and evaluate their expected information gain. We find that with a surprisingly modest resource budget, this simple Monte Carlo optimization strategy yields informative questions that mirror human performance across varied Battleship board scenarios. In contrast, LLM-only baselines struggle to ground questions in the board state; notably, GPT-4V provides no improvement over non-visual baselines. Our results illustrate how Bayesian models of question-asking can leverage the statistics of language to capture human priors, while highlighting some shortcomings of pure LLMs as grounded reasoners. |
| title | Loose LIPS Sink Ships: Asking Questions in Battleship with Language-Informed Program Sampling |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2402.19471 |