Learning, Fast and Slow: Towards LLMs That Adapt Continually
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909042868224000 |
|---|---|
| author | Tiwari, Rishabh Sareen, Kusha Agrawal, Lakshya A Gonzalez, Joseph E. Zaharia, Matei Keutzer, Kurt Dhillon, Inderjit S Agarwal, Rishabh Khatri, Devvrit |
| author_facet | Tiwari, Rishabh Sareen, Kusha Agrawal, Lakshya A Gonzalez, Joseph E. Zaharia, Matei Keutzer, Kurt Dhillon, Inderjit S Agarwal, Rishabh Khatri, Devvrit |
| contents | Large language models (LLMs) are trained for downstream tasks by updating their parameters (e.g., via RL). However, updating parameters forces them to absorb task-specific information, which can result in catastrophic forgetting and loss of plasticity. In contrast, in-context learning with fixed LLM parameters can cheaply and rapidly adapt to task-specific requirements (e.g., prompt optimization), but cannot by itself typically match the performance gains available through updating LLM parameters. There is no good reason for restricting learning to being in-context or in-weights. Moreover, humans also likely learn at different time scales (e.g., System 1 vs 2). To this end, we introduce a fast-slow learning framework for LLMs, with model parameters as "slow" weights and optimized context as "fast" weights. These fast "weights" can learn from textual feedback to absorb the task-specific information, while allowing slow weights to stay closer to the base model and persist general reasoning behaviors. Fast-Slow Training (FST) is up to 3x more sample-efficient than only slow learning (RL) across reasoning tasks, while consistently reaching a higher performance asymptote. Moreover, FST-trained models remain closer to the base LLM (up to 70% less KL divergence), resulting in less catastrophic forgetting than RL-training. This reduced drift also preserves plasticity: after training on one task, FST trained models adapt more effectively to a subsequent task than parameter-only trained models. In continual learning scenarios, where task domains change on the fly, FST continues to acquire each new task while parameter-only RL stalls. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_12484 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Learning, Fast and Slow: Towards LLMs That Adapt Continually Tiwari, Rishabh Sareen, Kusha Agrawal, Lakshya A Gonzalez, Joseph E. Zaharia, Matei Keutzer, Kurt Dhillon, Inderjit S Agarwal, Rishabh Khatri, Devvrit Machine Learning Artificial Intelligence I.2.6; I.2.7; I.2.8; I.2.4 Large language models (LLMs) are trained for downstream tasks by updating their parameters (e.g., via RL). However, updating parameters forces them to absorb task-specific information, which can result in catastrophic forgetting and loss of plasticity. In contrast, in-context learning with fixed LLM parameters can cheaply and rapidly adapt to task-specific requirements (e.g., prompt optimization), but cannot by itself typically match the performance gains available through updating LLM parameters. There is no good reason for restricting learning to being in-context or in-weights. Moreover, humans also likely learn at different time scales (e.g., System 1 vs 2). To this end, we introduce a fast-slow learning framework for LLMs, with model parameters as "slow" weights and optimized context as "fast" weights. These fast "weights" can learn from textual feedback to absorb the task-specific information, while allowing slow weights to stay closer to the base model and persist general reasoning behaviors. Fast-Slow Training (FST) is up to 3x more sample-efficient than only slow learning (RL) across reasoning tasks, while consistently reaching a higher performance asymptote. Moreover, FST-trained models remain closer to the base LLM (up to 70% less KL divergence), resulting in less catastrophic forgetting than RL-training. This reduced drift also preserves plasticity: after training on one task, FST trained models adapt more effectively to a subsequent task than parameter-only trained models. In continual learning scenarios, where task domains change on the fly, FST continues to acquire each new task while parameter-only RL stalls. |
| title | Learning, Fast and Slow: Towards LLMs That Adapt Continually |
| topic | Machine Learning Artificial Intelligence I.2.6; I.2.7; I.2.8; I.2.4 |
| url | https://arxiv.org/abs/2605.12484 |