Understanding LLMs' Fluid Intelligence Deficiency: An Analysis of the ARC Task
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929737681600512 |
|---|---|
| author | Wu, Junjie Yu, Mo Liu, Lemao Yeung, Dit-Yan Zhou, Jie |
| author_facet | Wu, Junjie Yu, Mo Liu, Lemao Yeung, Dit-Yan Zhou, Jie |
| contents | While LLMs have exhibited strong performance on various NLP tasks, it is noteworthy that most of these tasks rely on utilizing the vast amount of knowledge encoded in LLMs' parameters, rather than solving new problems without prior knowledge. In cognitive research, the latter ability is referred to as fluid intelligence, which is considered to be critical for assessing human intelligence. Recent research on fluid intelligence assessments has highlighted significant deficiencies in LLMs' abilities. In this paper, we analyze the challenges LLMs face in demonstrating fluid intelligence through controlled experiments, using the most representative ARC task as an example. Our study revealed three major limitations in existing LLMs: limited ability for skill composition, unfamiliarity with abstract input formats, and the intrinsic deficiency of left-to-right decoding. Our data and code can be found in https://wujunjie1998.github.io/araoc-benchmark.github.io/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_07190 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Understanding LLMs' Fluid Intelligence Deficiency: An Analysis of the ARC Task Wu, Junjie Yu, Mo Liu, Lemao Yeung, Dit-Yan Zhou, Jie Artificial Intelligence While LLMs have exhibited strong performance on various NLP tasks, it is noteworthy that most of these tasks rely on utilizing the vast amount of knowledge encoded in LLMs' parameters, rather than solving new problems without prior knowledge. In cognitive research, the latter ability is referred to as fluid intelligence, which is considered to be critical for assessing human intelligence. Recent research on fluid intelligence assessments has highlighted significant deficiencies in LLMs' abilities. In this paper, we analyze the challenges LLMs face in demonstrating fluid intelligence through controlled experiments, using the most representative ARC task as an example. Our study revealed three major limitations in existing LLMs: limited ability for skill composition, unfamiliarity with abstract input formats, and the intrinsic deficiency of left-to-right decoding. Our data and code can be found in https://wujunjie1998.github.io/araoc-benchmark.github.io/. |
| title | Understanding LLMs' Fluid Intelligence Deficiency: An Analysis of the ARC Task |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2502.07190 |