Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.01723 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915735014473728 |
|---|---|
| author | Oh, Soyoung Huang, Xinting Pink, Mathis Hahn, Michael Demberg, Vera |
| author_facet | Oh, Soyoung Huang, Xinting Pink, Mathis Hahn, Michael Demberg, Vera |
| contents | Idioms present a unique challenge for language models due to their non-compositional figurative interpretations, which often strongly diverge from the idiom's literal interpretation. In this paper, we employ causal tracing to systematically analyze how pretrained causal transformers deal with this ambiguity. We localize three mechanisms: (i) Early sublayers and specific attention heads retrieve an idiom's figurative interpretation, while suppressing its literal interpretation. (ii) When disambiguating context precedes the idiom, the model leverages it from the earliest layer and later layers refine the interpretation if the context conflicts with the retrieved interpretation. (iii) Then, selective, competing pathways carry both interpretations: an intermediate pathway prioritizes the figurative interpretation and a parallel direct route favors the literal interpretation, ensuring that both readings remain available. Our findings provide mechanistic evidence for idiom comprehension in autoregressive transformers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_01723 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Tug-of-war between idioms' figurative and literal interpretations in LLMs Oh, Soyoung Huang, Xinting Pink, Mathis Hahn, Michael Demberg, Vera Computation and Language Artificial Intelligence Idioms present a unique challenge for language models due to their non-compositional figurative interpretations, which often strongly diverge from the idiom's literal interpretation. In this paper, we employ causal tracing to systematically analyze how pretrained causal transformers deal with this ambiguity. We localize three mechanisms: (i) Early sublayers and specific attention heads retrieve an idiom's figurative interpretation, while suppressing its literal interpretation. (ii) When disambiguating context precedes the idiom, the model leverages it from the earliest layer and later layers refine the interpretation if the context conflicts with the retrieved interpretation. (iii) Then, selective, competing pathways carry both interpretations: an intermediate pathway prioritizes the figurative interpretation and a parallel direct route favors the literal interpretation, ensuring that both readings remain available. Our findings provide mechanistic evidence for idiom comprehension in autoregressive transformers. |
| title | Tug-of-war between idioms' figurative and literal interpretations in LLMs |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2506.01723 |