Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916116071186432 |
|---|---|
| author | Ramesh, Rahul Lubana, Ekdeep Singh Khona, Mikail Dick, Robert P. Tanaka, Hidenori |
| author_facet | Ramesh, Rahul Lubana, Ekdeep Singh Khona, Mikail Dick, Robert P. Tanaka, Hidenori |
| contents | Transformers trained on huge text corpora exhibit a remarkable set of capabilities, e.g., performing basic arithmetic. Given the inherent compositional nature of language, one can expect the model to learn to compose these capabilities, potentially yielding a combinatorial explosion of what operations it can perform on an input. Motivated by the above, we train autoregressive Transformer models on a synthetic data-generating process that involves compositions of a set of well-defined monolithic capabilities. Through a series of extensive and systematic experiments on this data-generating process, we show that: (1) autoregressive Transformers can learn compositional structures from small amounts of training data and generalize to exponentially or even combinatorially many functions; (2) generating intermediate outputs when composing functions is more effective for generalizing to new, unseen compositions than not generating any intermediate outputs (3) biases in the order of the compositions in the training data result in Transformers that fail to compose some combinations of functions; and (4) the attention layers select which capability to apply while the feed-forward layers execute the selected capability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_12997 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks Ramesh, Rahul Lubana, Ekdeep Singh Khona, Mikail Dick, Robert P. Tanaka, Hidenori Machine Learning Transformers trained on huge text corpora exhibit a remarkable set of capabilities, e.g., performing basic arithmetic. Given the inherent compositional nature of language, one can expect the model to learn to compose these capabilities, potentially yielding a combinatorial explosion of what operations it can perform on an input. Motivated by the above, we train autoregressive Transformer models on a synthetic data-generating process that involves compositions of a set of well-defined monolithic capabilities. Through a series of extensive and systematic experiments on this data-generating process, we show that: (1) autoregressive Transformers can learn compositional structures from small amounts of training data and generalize to exponentially or even combinatorially many functions; (2) generating intermediate outputs when composing functions is more effective for generalizing to new, unseen compositions than not generating any intermediate outputs (3) biases in the order of the compositions in the training data result in Transformers that fail to compose some combinations of functions; and (4) the attention layers select which capability to apply while the feed-forward layers execute the selected capability. |
| title | Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2311.12997 |