CLEAR: Can Language Models Really Understand Causal Graphs?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929397760524288 |
|---|---|
| author | Chen, Sirui Xu, Mengying Wang, Kun Zeng, Xingyu Zhao, Rui Zhao, Shengjie Lu, Chaochao |
| author_facet | Chen, Sirui Xu, Mengying Wang, Kun Zeng, Xingyu Zhao, Rui Zhao, Shengjie Lu, Chaochao |
| contents | Causal reasoning is a cornerstone of how humans interpret the world. To model and reason about causality, causal graphs offer a concise yet effective solution. Given the impressive advancements in language models, a crucial question arises: can they really understand causal graphs? To this end, we pioneer an investigation into language models' understanding of causal graphs. Specifically, we develop a framework to define causal graph understanding, by assessing language models' behaviors through four practical criteria derived from diverse disciplines (e.g., philosophy and psychology). We then develop CLEAR, a novel benchmark that defines three complexity levels and encompasses 20 causal graph-based tasks across these levels. Finally, based on our framework and benchmark, we conduct extensive experiments on six leading language models and summarize five empirical findings. Our results indicate that while language models demonstrate a preliminary understanding of causal graphs, significant potential for improvement remains. Our project website is at https://github.com/OpenCausaLab/CLEAR. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_16605 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CLEAR: Can Language Models Really Understand Causal Graphs? Chen, Sirui Xu, Mengying Wang, Kun Zeng, Xingyu Zhao, Rui Zhao, Shengjie Lu, Chaochao Computation and Language Artificial Intelligence Machine Learning Methodology Causal reasoning is a cornerstone of how humans interpret the world. To model and reason about causality, causal graphs offer a concise yet effective solution. Given the impressive advancements in language models, a crucial question arises: can they really understand causal graphs? To this end, we pioneer an investigation into language models' understanding of causal graphs. Specifically, we develop a framework to define causal graph understanding, by assessing language models' behaviors through four practical criteria derived from diverse disciplines (e.g., philosophy and psychology). We then develop CLEAR, a novel benchmark that defines three complexity levels and encompasses 20 causal graph-based tasks across these levels. Finally, based on our framework and benchmark, we conduct extensive experiments on six leading language models and summarize five empirical findings. Our results indicate that while language models demonstrate a preliminary understanding of causal graphs, significant potential for improvement remains. Our project website is at https://github.com/OpenCausaLab/CLEAR. |
| title | CLEAR: Can Language Models Really Understand Causal Graphs? |
| topic | Computation and Language Artificial Intelligence Machine Learning Methodology |
| url | https://arxiv.org/abs/2406.16605 |