When can transformers compositionally generalize in-context?
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916328111079424 |
|---|---|
| author | Kobayashi, Seijin Schug, Simon Akram, Yassir Redhardt, Florian von Oswald, Johannes Pascanu, Razvan Lajoie, Guillaume Sacramento, João |
| author_facet | Kobayashi, Seijin Schug, Simon Akram, Yassir Redhardt, Florian von Oswald, Johannes Pascanu, Razvan Lajoie, Guillaume Sacramento, João |
| contents | Many tasks can be composed from a few independent components. This gives rise to a combinatorial explosion of possible tasks, only some of which might be encountered during training. Under what circumstances can transformers compositionally generalize from a subset of tasks to all possible combinations of tasks that share similar components? Here we study a modular multitask setting that allows us to precisely control compositional structure in the data generation process. We present evidence that transformers learning in-context struggle to generalize compositionally on this task despite being in principle expressive enough to do so. Compositional generalization becomes possible only when introducing a bottleneck that enforces an explicit separation between task inference and task execution. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_12275 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | When can transformers compositionally generalize in-context? Kobayashi, Seijin Schug, Simon Akram, Yassir Redhardt, Florian von Oswald, Johannes Pascanu, Razvan Lajoie, Guillaume Sacramento, João Machine Learning Neural and Evolutionary Computing Many tasks can be composed from a few independent components. This gives rise to a combinatorial explosion of possible tasks, only some of which might be encountered during training. Under what circumstances can transformers compositionally generalize from a subset of tasks to all possible combinations of tasks that share similar components? Here we study a modular multitask setting that allows us to precisely control compositional structure in the data generation process. We present evidence that transformers learning in-context struggle to generalize compositionally on this task despite being in principle expressive enough to do so. Compositional generalization becomes possible only when introducing a bottleneck that enforces an explicit separation between task inference and task execution. |
| title | When can transformers compositionally generalize in-context? |
| topic | Machine Learning Neural and Evolutionary Computing |
| url | https://arxiv.org/abs/2407.12275 |