Evidence from counterfactual tasks supports emergent analogical reasoning in large language models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913335527604224 |
|---|---|
| author | Webb, Taylor Holyoak, Keith J. Lu, Hongjing |
| author_facet | Webb, Taylor Holyoak, Keith J. Lu, Hongjing |
| contents | We recently reported evidence that large language models are capable of solving a wide range of text-based analogy problems in a zero-shot manner, indicating the presence of an emergent capacity for analogical reasoning. Two recent commentaries have challenged these results, citing evidence from so-called `counterfactual' tasks in which the standard sequence of the alphabet is arbitrarily permuted so as to decrease similarity with materials that may have been present in the language model's training data. Here, we reply to these critiques, clarifying some misunderstandings about the test materials used in our original work, and presenting evidence that language models are also capable of generalizing to these new counterfactual task variants. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_13070 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Evidence from counterfactual tasks supports emergent analogical reasoning in large language models Webb, Taylor Holyoak, Keith J. Lu, Hongjing Computation and Language Artificial Intelligence We recently reported evidence that large language models are capable of solving a wide range of text-based analogy problems in a zero-shot manner, indicating the presence of an emergent capacity for analogical reasoning. Two recent commentaries have challenged these results, citing evidence from so-called `counterfactual' tasks in which the standard sequence of the alphabet is arbitrarily permuted so as to decrease similarity with materials that may have been present in the language model's training data. Here, we reply to these critiques, clarifying some misunderstandings about the test materials used in our original work, and presenting evidence that language models are also capable of generalizing to these new counterfactual task variants. |
| title | Evidence from counterfactual tasks supports emergent analogical reasoning in large language models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2404.13070 |