Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914600222457856 |
|---|---|
| author | Marek, Martin Cho, Dongkyu Qiu, Shikai Chunara, Rumi Izmailov, Pavel Wilson, Andrew Gordon |
| author_facet | Marek, Martin Cho, Dongkyu Qiu, Shikai Chunara, Rumi Izmailov, Pavel Wilson, Andrew Gordon |
| contents | Models trained on a new task typically degrade on prior tasks, a phenomenon known as forgetting. Traditionally, mitigating forgetting has required replaying stored exemplars from prior tasks, which is often impractical. By contrast, language models can sample from their own training distribution, and we show that these self-generated samples serve as effective replay data, nearly eliminating forgetting. We find that forgetting nonetheless persists when the model has little remaining capacity: models pretrained close to saturation cannot absorb new information without overwriting prior knowledge. When capacity is not the limiting factor, low learning rates reduce forgetting but require substantially more training steps. Replay breaks this tradeoff, enabling fast, high-learning-rate finetuning without forgetting. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_26097 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay Marek, Martin Cho, Dongkyu Qiu, Shikai Chunara, Rumi Izmailov, Pavel Wilson, Andrew Gordon Machine Learning Models trained on a new task typically degrade on prior tasks, a phenomenon known as forgetting. Traditionally, mitigating forgetting has required replaying stored exemplars from prior tasks, which is often impractical. By contrast, language models can sample from their own training distribution, and we show that these self-generated samples serve as effective replay data, nearly eliminating forgetting. We find that forgetting nonetheless persists when the model has little remaining capacity: models pretrained close to saturation cannot absorb new information without overwriting prior knowledge. When capacity is not the limiting factor, low learning rates reduce forgetting but require substantially more training steps. Replay breaks this tradeoff, enabling fast, high-learning-rate finetuning without forgetting. |
| title | Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2605.26097 |