Linear Transformers with Learnable Kernel Functions are Better In-Context Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913378527608832 |
|---|---|
| author | Aksenov, Yaroslav Balagansky, Nikita Vaina, Sofia Maria Lo Cicero Shaposhnikov, Boris Gorbatovski, Alexey Gavrilov, Daniil |
| author_facet | Aksenov, Yaroslav Balagansky, Nikita Vaina, Sofia Maria Lo Cicero Shaposhnikov, Boris Gorbatovski, Alexey Gavrilov, Daniil |
| contents | Advancing the frontier of subquadratic architectures for Language Models (LMs) is crucial in the rapidly evolving field of natural language processing. Current innovations, including State Space Models, were initially celebrated for surpassing Transformer performance on language modeling tasks. However, these models have revealed deficiencies in essential In-Context Learning capabilities - a domain where the Transformer traditionally shines. The Based model emerged as a hybrid solution, blending a Linear Transformer with a kernel inspired by the Taylor expansion of exponential functions, augmented by convolutional networks. Mirroring the Transformer's in-context adeptness, it became a strong contender in the field. In our work, we present a singular, elegant alteration to the Based kernel that amplifies its In-Context Learning abilities evaluated with the Multi-Query Associative Recall task and overall language modeling process, as demonstrated on the Pile dataset. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_10644 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Linear Transformers with Learnable Kernel Functions are Better In-Context Models Aksenov, Yaroslav Balagansky, Nikita Vaina, Sofia Maria Lo Cicero Shaposhnikov, Boris Gorbatovski, Alexey Gavrilov, Daniil Machine Learning Computation and Language Advancing the frontier of subquadratic architectures for Language Models (LMs) is crucial in the rapidly evolving field of natural language processing. Current innovations, including State Space Models, were initially celebrated for surpassing Transformer performance on language modeling tasks. However, these models have revealed deficiencies in essential In-Context Learning capabilities - a domain where the Transformer traditionally shines. The Based model emerged as a hybrid solution, blending a Linear Transformer with a kernel inspired by the Taylor expansion of exponential functions, augmented by convolutional networks. Mirroring the Transformer's in-context adeptness, it became a strong contender in the field. In our work, we present a singular, elegant alteration to the Based kernel that amplifies its In-Context Learning abilities evaluated with the Multi-Query Associative Recall task and overall language modeling process, as demonstrated on the Pile dataset. |
| title | Linear Transformers with Learnable Kernel Functions are Better In-Context Models |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2402.10644 |