Traffic Light or Light Traffic? Investigating Phrasal Semantics in Large Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914963469107200 |
|---|---|
| author | Meng, Rui Liu, Ye Tu, Lifu He, Daqing Zhou, Yingbo Yavuz, Semih |
| author_facet | Meng, Rui Liu, Ye Tu, Lifu He, Daqing Zhou, Yingbo Yavuz, Semih |
| contents | Phrases are fundamental linguistic units through which humans convey semantics. This study critically examines the capacity of API-based large language models (LLMs) to comprehend phrase semantics, utilizing three human-annotated datasets. We assess the performance of LLMs in executing phrase semantic reasoning tasks guided by natural language instructions and explore the impact of common prompting techniques, including few-shot demonstrations and Chain-of-Thought reasoning. Our findings reveal that LLMs greatly outperform traditional embedding methods across the datasets; however, they do not show a significant advantage over fine-tuned methods. The effectiveness of advanced prompting strategies shows variability. We conduct detailed error analyses to interpret the limitations faced by LLMs in comprehending phrase semantics. Code and data can be found at https://github.com/memray/llm_phrase_semantics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_02308 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Traffic Light or Light Traffic? Investigating Phrasal Semantics in Large Language Models Meng, Rui Liu, Ye Tu, Lifu He, Daqing Zhou, Yingbo Yavuz, Semih Computation and Language Phrases are fundamental linguistic units through which humans convey semantics. This study critically examines the capacity of API-based large language models (LLMs) to comprehend phrase semantics, utilizing three human-annotated datasets. We assess the performance of LLMs in executing phrase semantic reasoning tasks guided by natural language instructions and explore the impact of common prompting techniques, including few-shot demonstrations and Chain-of-Thought reasoning. Our findings reveal that LLMs greatly outperform traditional embedding methods across the datasets; however, they do not show a significant advantage over fine-tuned methods. The effectiveness of advanced prompting strategies shows variability. We conduct detailed error analyses to interpret the limitations faced by LLMs in comprehending phrase semantics. Code and data can be found at https://github.com/memray/llm_phrase_semantics. |
| title | Traffic Light or Light Traffic? Investigating Phrasal Semantics in Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.02308 |