Can Large Language Models Learn Independent Causal Mechanisms?
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910596803330048 |
|---|---|
| author | Gendron, Gaël Nguyen, Bao Trung Peng, Alex Yuxuan Witbrock, Michael Dobbie, Gillian |
| author_facet | Gendron, Gaël Nguyen, Bao Trung Peng, Alex Yuxuan Witbrock, Michael Dobbie, Gillian |
| contents | Despite impressive performance on language modelling and complex reasoning tasks, Large Language Models (LLMs) fall short on the same tasks in uncommon settings or with distribution shifts, exhibiting a lack of generalisation ability. By contrast, systems such as causal models, that learn abstract variables and causal relationships, can demonstrate increased robustness against changes in the distribution. One reason for this success is the existence and use of Independent Causal Mechanisms (ICMs) representing high-level concepts that only sparsely interact. In this work, we apply two concepts from causality to learn ICMs within LLMs. We develop a new LLM architecture composed of multiple sparsely interacting language modelling modules. We show that such causal constraints can improve out-of-distribution performance on abstract and causal reasoning tasks. We also investigate the level of independence and domain specialisation and show that LLMs rely on pre-trained partially domain-invariant mechanisms resilient to fine-tuning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_02636 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Can Large Language Models Learn Independent Causal Mechanisms? Gendron, Gaël Nguyen, Bao Trung Peng, Alex Yuxuan Witbrock, Michael Dobbie, Gillian Computation and Language Artificial Intelligence Information Theory Machine Learning I.2.3; I.2.6; I.2.7; G.3 Despite impressive performance on language modelling and complex reasoning tasks, Large Language Models (LLMs) fall short on the same tasks in uncommon settings or with distribution shifts, exhibiting a lack of generalisation ability. By contrast, systems such as causal models, that learn abstract variables and causal relationships, can demonstrate increased robustness against changes in the distribution. One reason for this success is the existence and use of Independent Causal Mechanisms (ICMs) representing high-level concepts that only sparsely interact. In this work, we apply two concepts from causality to learn ICMs within LLMs. We develop a new LLM architecture composed of multiple sparsely interacting language modelling modules. We show that such causal constraints can improve out-of-distribution performance on abstract and causal reasoning tasks. We also investigate the level of independence and domain specialisation and show that LLMs rely on pre-trained partially domain-invariant mechanisms resilient to fine-tuning. |
| title | Can Large Language Models Learn Independent Causal Mechanisms? |
| topic | Computation and Language Artificial Intelligence Information Theory Machine Learning I.2.3; I.2.6; I.2.7; G.3 |
| url | https://arxiv.org/abs/2402.02636 |