Do Attention Heads Compete or Cooperate during Counting?
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917918494687232 |
|---|---|
| author | Zsámboki, Pál Fraknói, Ádám Gedeon, Máté Kornai, András Zombori, Zsolt |
| author_facet | Zsámboki, Pál Fraknói, Ádám Gedeon, Máté Kornai, András Zombori, Zsolt |
| contents | We present an in-depth mechanistic interpretability analysis of training small transformers on an elementary task, counting, which is a crucial deductive step in many algorithms. In particular, we investigate the collaboration/competition among the attention heads: we ask whether the attention heads behave as a pseudo-ensemble, all solving the same subtask, or they perform different subtasks, meaning that they can only solve the original task in conjunction. Our work presents evidence that on the semantics of the counting task, attention heads behave as a pseudo-ensemble, but their outputs need to be aggregated in a non-uniform manner in order to create an encoding that conforms to the syntax. Our source code will be available upon publication. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_06923 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Do Attention Heads Compete or Cooperate during Counting? Zsámboki, Pál Fraknói, Ádám Gedeon, Máté Kornai, András Zombori, Zsolt Machine Learning Artificial Intelligence We present an in-depth mechanistic interpretability analysis of training small transformers on an elementary task, counting, which is a crucial deductive step in many algorithms. In particular, we investigate the collaboration/competition among the attention heads: we ask whether the attention heads behave as a pseudo-ensemble, all solving the same subtask, or they perform different subtasks, meaning that they can only solve the original task in conjunction. Our work presents evidence that on the semantics of the counting task, attention heads behave as a pseudo-ensemble, but their outputs need to be aggregated in a non-uniform manner in order to create an encoding that conforms to the syntax. Our source code will be available upon publication. |
| title | Do Attention Heads Compete or Cooperate during Counting? |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2502.06923 |