A Meta-Learning Perspective on Transformers for Causal Language Modeling
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913280583270400 |
|---|---|
| author | Wu, Xinbo Varshney, Lav R. |
| author_facet | Wu, Xinbo Varshney, Lav R. |
| contents | The Transformer architecture has become prominent in developing large causal language models. However, mechanisms to explain its capabilities are not well understood. Focused on the training process, here we establish a meta-learning view of the Transformer architecture when trained for the causal language modeling task, by explicating an inner optimization process within the Transformer. Further, within the inner optimization, we discover and theoretically analyze a special characteristic of the norms of learned token representations within Transformer-based causal language models. Our analysis is supported by experiments in various settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_05884 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | A Meta-Learning Perspective on Transformers for Causal Language Modeling Wu, Xinbo Varshney, Lav R. Machine Learning Artificial Intelligence Computation and Language The Transformer architecture has become prominent in developing large causal language models. However, mechanisms to explain its capabilities are not well understood. Focused on the training process, here we establish a meta-learning view of the Transformer architecture when trained for the causal language modeling task, by explicating an inner optimization process within the Transformer. Further, within the inner optimization, we discover and theoretically analyze a special characteristic of the norms of learned token representations within Transformer-based causal language models. Our analysis is supported by experiments in various settings. |
| title | A Meta-Learning Perspective on Transformers for Causal Language Modeling |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2310.05884 |