Assessing Logical Reasoning Capabilities of Encoder-Only Transformer Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929404803809280 |
|---|---|
| author | Pirozelli, Paulo José, Marcos M. Filho, Paulo de Tarso P. Brandão, Anarosa A. F. Cozman, Fabio G. |
| author_facet | Pirozelli, Paulo José, Marcos M. Filho, Paulo de Tarso P. Brandão, Anarosa A. F. Cozman, Fabio G. |
| contents | Logical reasoning is central to complex human activities, such as thinking, debating, and planning; it is also a central component of many AI systems as well. In this paper, we investigate the extent to which encoder-only transformer language models (LMs) can reason according to logical rules. We ask whether those LMs can deduce theorems in propositional calculus and first-order logic; if their relative success in these problems reflects general logical capabilities; and which layers contribute the most to the task. First, we show for several encoder-only LMs that they can be trained, to a reasonable degree, to determine logical validity on various datasets. Next, by cross-probing fine-tuned models on these datasets, we show that LMs have difficulty in transferring their putative logical reasoning ability, which suggests that they may have learned dataset-specific features, instead of a general capability. Finally, we conduct a layerwise probing experiment, which shows that the hypothesis classification task is mostly solved through higher layers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_11720 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Assessing Logical Reasoning Capabilities of Encoder-Only Transformer Models Pirozelli, Paulo José, Marcos M. Filho, Paulo de Tarso P. Brandão, Anarosa A. F. Cozman, Fabio G. Computation and Language Artificial Intelligence Logical reasoning is central to complex human activities, such as thinking, debating, and planning; it is also a central component of many AI systems as well. In this paper, we investigate the extent to which encoder-only transformer language models (LMs) can reason according to logical rules. We ask whether those LMs can deduce theorems in propositional calculus and first-order logic; if their relative success in these problems reflects general logical capabilities; and which layers contribute the most to the task. First, we show for several encoder-only LMs that they can be trained, to a reasonable degree, to determine logical validity on various datasets. Next, by cross-probing fine-tuned models on these datasets, we show that LMs have difficulty in transferring their putative logical reasoning ability, which suggests that they may have learned dataset-specific features, instead of a general capability. Finally, we conduct a layerwise probing experiment, which shows that the hypothesis classification task is mostly solved through higher layers. |
| title | Assessing Logical Reasoning Capabilities of Encoder-Only Transformer Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2312.11720 |