Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.23006 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914354956337152 |
|---|---|
| author | Wang, Shenran Tse, Timothy Tin-Long Zhu, Jian |
| author_facet | Wang, Shenran Tse, Timothy Tin-Long Zhu, Jian |
| contents | We perform in-depth evaluations of in-context learning (ICL) on state-of-the-art transformer, state-space, and hybrid large language models over two categories of knowledge-based ICL tasks. Using a combination of behavioral probing and intervention-based methods, we have discovered that, while LLMs of different architectures can behave similarly in task performance, their internals could remain different. We discover that function vectors (FVs) responsible for ICL are primarily located in the self-attention and Mamba layers, and speculate that Mamba2 uses a different mechanism from FVs to perform ICL. FVs are more important for ICL involving parametric knowledge retrieval, but not for contextual knowledge understanding. Our work contributes to a more nuanced understanding across architectures and task types. Methodologically, our approach also highlights the importance of combining both behavioural and mechanistic analyses to investigate LLM capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_23006 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Understanding In-Context Learning Beyond Transformers: An Investigation of State Space and Hybrid Architectures Wang, Shenran Tse, Timothy Tin-Long Zhu, Jian Computation and Language Artificial Intelligence We perform in-depth evaluations of in-context learning (ICL) on state-of-the-art transformer, state-space, and hybrid large language models over two categories of knowledge-based ICL tasks. Using a combination of behavioral probing and intervention-based methods, we have discovered that, while LLMs of different architectures can behave similarly in task performance, their internals could remain different. We discover that function vectors (FVs) responsible for ICL are primarily located in the self-attention and Mamba layers, and speculate that Mamba2 uses a different mechanism from FVs to perform ICL. FVs are more important for ICL involving parametric knowledge retrieval, but not for contextual knowledge understanding. Our work contributes to a more nuanced understanding across architectures and task types. Methodologically, our approach also highlights the importance of combining both behavioural and mechanistic analyses to investigate LLM capabilities. |
| title | Understanding In-Context Learning Beyond Transformers: An Investigation of State Space and Hybrid Architectures |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2510.23006 |