Understanding and Controlling Repetition Neurons and Induction Heads in In-Context Learning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915611479638016 |
|---|---|
| author | Doan, Nhi Hoai Hiraoka, Tatsuya Inui, Kentaro |
| author_facet | Doan, Nhi Hoai Hiraoka, Tatsuya Inui, Kentaro |
| contents | This paper investigates the relationship between large language models' (LLMs) ability to recognize repetitive input patterns and their performance on in-context learning (ICL). In contrast to prior work that has primarily focused on attention heads, we examine this relationship from the perspective of skill neurons, specifically repetition neurons. Our experiments reveal that the impact of these neurons on ICL performance varies depending on the depth of the layer in which they reside. By comparing the effects of repetition neurons and induction heads, we further identify strategies for reducing repetitive outputs while maintaining strong ICL capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_07810 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Understanding and Controlling Repetition Neurons and Induction Heads in In-Context Learning Doan, Nhi Hoai Hiraoka, Tatsuya Inui, Kentaro Computation and Language This paper investigates the relationship between large language models' (LLMs) ability to recognize repetitive input patterns and their performance on in-context learning (ICL). In contrast to prior work that has primarily focused on attention heads, we examine this relationship from the perspective of skill neurons, specifically repetition neurons. Our experiments reveal that the impact of these neurons on ICL performance varies depending on the depth of the layer in which they reside. By comparing the effects of repetition neurons and induction heads, we further identify strategies for reducing repetitive outputs while maintaining strong ICL capabilities. |
| title | Understanding and Controlling Repetition Neurons and Induction Heads in In-Context Learning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2507.07810 |