State Soup: In-Context Skill Learning, Retrieval and Mixing
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909222500827136 |
|---|---|
| author | Pióro, Maciej Wołczyk, Maciej Pascanu, Razvan von Oswald, Johannes Sacramento, João |
| author_facet | Pióro, Maciej Wołczyk, Maciej Pascanu, Razvan von Oswald, Johannes Sacramento, João |
| contents | A new breed of gated-linear recurrent neural networks has reached state-of-the-art performance on a range of sequence modeling problems. Such models naturally handle long sequences efficiently, as the cost of processing a new input is independent of sequence length. Here, we explore another advantage of these stateful sequence models, inspired by the success of model merging through parameter interpolation. Building on parallels between fine-tuning and in-context learning, we investigate whether we can treat internal states as task vectors that can be stored, retrieved, and then linearly combined, exploiting the linearity of recurrence. We study this form of fast model merging on Mamba-2.8b, a pretrained recurrent model, and present preliminary evidence that simple linear state interpolation methods suffice to improve next-token perplexity as well as downstream in-context learning task performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_08423 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | State Soup: In-Context Skill Learning, Retrieval and Mixing Pióro, Maciej Wołczyk, Maciej Pascanu, Razvan von Oswald, Johannes Sacramento, João Machine Learning Artificial Intelligence A new breed of gated-linear recurrent neural networks has reached state-of-the-art performance on a range of sequence modeling problems. Such models naturally handle long sequences efficiently, as the cost of processing a new input is independent of sequence length. Here, we explore another advantage of these stateful sequence models, inspired by the success of model merging through parameter interpolation. Building on parallels between fine-tuning and in-context learning, we investigate whether we can treat internal states as task vectors that can be stored, retrieved, and then linearly combined, exploiting the linearity of recurrence. We study this form of fast model merging on Mamba-2.8b, a pretrained recurrent model, and present preliminary evidence that simple linear state interpolation methods suffice to improve next-token perplexity as well as downstream in-context learning task performance. |
| title | State Soup: In-Context Skill Learning, Retrieval and Mixing |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2406.08423 |