AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909335032954880 |
|---|---|
| author | Ye, Xiao Wang, Andrew Choi, Jacob Lu, Yining Sharma, Shreya Shen, Lingfeng Tiyyala, Vijay Andrews, Nicholas Khashabi, Daniel |
| author_facet | Ye, Xiao Wang, Andrew Choi, Jacob Lu, Yining Sharma, Shreya Shen, Lingfeng Tiyyala, Vijay Andrews, Nicholas Khashabi, Daniel |
| contents | Humans regularly engage in analogical thinking, relating personal experiences to current situations (X is analogous to Y because of Z). Analogical thinking allows humans to solve problems in creative ways, grasp difficult concepts, and articulate ideas more effectively. Can language models (LMs) do the same? To answer this question, we propose AnaloBench, a benchmark to determine analogical reasoning ability in LMs. Our benchmarking approach focuses on aspects of this ability that are common among humans: (i) recalling related experiences from a large amount of information, and (ii) applying analogical reasoning to complex and lengthy scenarios. We test a broad collection of proprietary models (e.g., GPT family, Claude V2) and open source models such as LLaMA2. As in prior results, scaling up LMs results in some performance boosts. Surprisingly, scale offers minimal gains when, (i) analogies involve lengthy scenarios, or (ii) recalling relevant scenarios from a large pool of information, a process analogous to finding a needle in a haystack. We hope these observations encourage further research in this field. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_12370 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies Ye, Xiao Wang, Andrew Choi, Jacob Lu, Yining Sharma, Shreya Shen, Lingfeng Tiyyala, Vijay Andrews, Nicholas Khashabi, Daniel Computation and Language Artificial Intelligence Humans regularly engage in analogical thinking, relating personal experiences to current situations (X is analogous to Y because of Z). Analogical thinking allows humans to solve problems in creative ways, grasp difficult concepts, and articulate ideas more effectively. Can language models (LMs) do the same? To answer this question, we propose AnaloBench, a benchmark to determine analogical reasoning ability in LMs. Our benchmarking approach focuses on aspects of this ability that are common among humans: (i) recalling related experiences from a large amount of information, and (ii) applying analogical reasoning to complex and lengthy scenarios. We test a broad collection of proprietary models (e.g., GPT family, Claude V2) and open source models such as LLaMA2. As in prior results, scaling up LMs results in some performance boosts. Surprisingly, scale offers minimal gains when, (i) analogies involve lengthy scenarios, or (ii) recalling relevant scenarios from a large pool of information, a process analogous to finding a needle in a haystack. We hope these observations encourage further research in this field. |
| title | AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2402.12370 |