Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2403.17760 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916265571909632 |
|---|---|
| author | Zhou, Shijia Weissweiler, Leonie He, Taiqi Schütze, Hinrich Mortensen, David R. Levin, Lori |
| author_facet | Zhou, Shijia Weissweiler, Leonie He, Taiqi Schütze, Hinrich Mortensen, David R. Levin, Lori |
| contents | In this paper, we make a contribution that can be understood from two perspectives: from an NLP perspective, we introduce a small challenge dataset for NLI with large lexical overlap, which minimises the possibility of models discerning entailment solely based on token distinctions, and show that GPT-4 and Llama 2 fail it with strong bias. We then create further challenging sub-tasks in an effort to explain this failure. From a Computational Linguistics perspective, we identify a group of constructions with three classes of adjectives which cannot be distinguished by surface features. This enables us to probe for LLM's understanding of these constructions in various ways, and we find that they fail in a variety of ways to distinguish between them, suggesting that they don't adequately represent their meaning or capture the lexical properties of phrasal heads. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_17760 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasons Zhou, Shijia Weissweiler, Leonie He, Taiqi Schütze, Hinrich Mortensen, David R. Levin, Lori Computation and Language In this paper, we make a contribution that can be understood from two perspectives: from an NLP perspective, we introduce a small challenge dataset for NLI with large lexical overlap, which minimises the possibility of models discerning entailment solely based on token distinctions, and show that GPT-4 and Llama 2 fail it with strong bias. We then create further challenging sub-tasks in an effort to explain this failure. From a Computational Linguistics perspective, we identify a group of constructions with three classes of adjectives which cannot be distinguished by surface features. This enables us to probe for LLM's understanding of these constructions in various ways, and we find that they fail in a variety of ways to distinguish between them, suggesting that they don't adequately represent their meaning or capture the lexical properties of phrasal heads. |
| title | Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasons |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2403.17760 |