Exploring Forgetting in Large Language Model Pre-Training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914984086208512 |
|---|---|
| author | Liao, Chonghua Xie, Ruobing Sun, Xingwu Sun, Haowen Kang, Zhanhui |
| author_facet | Liao, Chonghua Xie, Ruobing Sun, Xingwu Sun, Haowen Kang, Zhanhui |
| contents | Catastrophic forgetting remains a formidable obstacle to building an omniscient model in large language models (LLMs). Despite the pioneering research on task-level forgetting in LLM fine-tuning, there is scant focus on forgetting during pre-training. We systematically explored the existence and measurement of forgetting in pre-training, questioning traditional metrics such as perplexity (PPL) and introducing new metrics to better detect entity memory retention. Based on our revised assessment of forgetting metrics, we explored low-cost, straightforward methods to mitigate forgetting during the pre-training phase. Further, we carefully analyzed the learning curves, offering insights into the dynamics of forgetting. Extensive evaluations and analyses on forgetting of pre-training could facilitate future research on LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_17018 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Exploring Forgetting in Large Language Model Pre-Training Liao, Chonghua Xie, Ruobing Sun, Xingwu Sun, Haowen Kang, Zhanhui Computation and Language Catastrophic forgetting remains a formidable obstacle to building an omniscient model in large language models (LLMs). Despite the pioneering research on task-level forgetting in LLM fine-tuning, there is scant focus on forgetting during pre-training. We systematically explored the existence and measurement of forgetting in pre-training, questioning traditional metrics such as perplexity (PPL) and introducing new metrics to better detect entity memory retention. Based on our revised assessment of forgetting metrics, we explored low-cost, straightforward methods to mitigate forgetting during the pre-training phase. Further, we carefully analyzed the learning curves, offering insights into the dynamics of forgetting. Extensive evaluations and analyses on forgetting of pre-training could facilitate future research on LLMs. |
| title | Exploring Forgetting in Large Language Model Pre-Training |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.17018 |