ExpliCa: Evaluating Explicit Causal Reasoning in Large Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914315072700416 |
|---|---|
| author | Miliani, Martina Auriemma, Serena Bondielli, Alessandro Chersoni, Emmanuele Passaro, Lucia Sucameli, Irene Lenci, Alessandro |
| author_facet | Miliani, Martina Auriemma, Serena Bondielli, Alessandro Chersoni, Emmanuele Passaro, Lucia Sucameli, Irene Lenci, Alessandro |
| contents | Large Language Models (LLMs) are increasingly used in tasks requiring interpretive and inferential accuracy. In this paper, we introduce ExpliCa, a new dataset for evaluating LLMs in explicit causal reasoning. ExpliCa uniquely integrates both causal and temporal relations presented in different linguistic orders and explicitly expressed by linguistic connectives. The dataset is enriched with crowdsourced human acceptability ratings. We tested LLMs on ExpliCa through prompting and perplexity-based metrics. We assessed seven commercial and open-source LLMs, revealing that even top models struggle to reach 0.80 accuracy. Interestingly, models tend to confound temporal relations with causal ones, and their performance is also strongly influenced by the linguistic order of the events. Finally, perplexity-based scores and prompting performance are differently affected by model size. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_15487 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ExpliCa: Evaluating Explicit Causal Reasoning in Large Language Models Miliani, Martina Auriemma, Serena Bondielli, Alessandro Chersoni, Emmanuele Passaro, Lucia Sucameli, Irene Lenci, Alessandro Computation and Language Artificial Intelligence 68T50, 68T07 I.2.7 Large Language Models (LLMs) are increasingly used in tasks requiring interpretive and inferential accuracy. In this paper, we introduce ExpliCa, a new dataset for evaluating LLMs in explicit causal reasoning. ExpliCa uniquely integrates both causal and temporal relations presented in different linguistic orders and explicitly expressed by linguistic connectives. The dataset is enriched with crowdsourced human acceptability ratings. We tested LLMs on ExpliCa through prompting and perplexity-based metrics. We assessed seven commercial and open-source LLMs, revealing that even top models struggle to reach 0.80 accuracy. Interestingly, models tend to confound temporal relations with causal ones, and their performance is also strongly influenced by the linguistic order of the events. Finally, perplexity-based scores and prompting performance are differently affected by model size. |
| title | ExpliCa: Evaluating Explicit Causal Reasoning in Large Language Models |
| topic | Computation and Language Artificial Intelligence 68T50, 68T07 I.2.7 |
| url | https://arxiv.org/abs/2502.15487 |