TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916781620199424 |
|---|---|
| author | Li, Jeffrey Armandpour, Mohammadreza Mirzadeh, Iman Mehta, Sachin Shankar, Vaishaal Vemulapalli, Raviteja Bengio, Samy Tuzel, Oncel Farajtabar, Mehrdad Pouransari, Hadi Faghri, Fartash |
| author_facet | Li, Jeffrey Armandpour, Mohammadreza Mirzadeh, Iman Mehta, Sachin Shankar, Vaishaal Vemulapalli, Raviteja Bengio, Samy Tuzel, Oncel Farajtabar, Mehrdad Pouransari, Hadi Faghri, Fartash |
| contents | Large Language Models (LLMs) trained on historical web data inevitably become outdated. We investigate evaluation strategies and update methods for LLMs as new data becomes available. We introduce a web-scale dataset for time-continual pretraining of LLMs derived from 114 dumps of Common Crawl (CC) - orders of magnitude larger than previous continual language modeling benchmarks. We also design time-stratified evaluations across both general CC data and specific domains (Wikipedia, StackExchange, and code documentation) to assess how well various continual learning methods adapt to new data while retaining past knowledge. Our findings demonstrate that, on general CC data, autoregressive meta-schedules combined with a fixed-ratio replay of older data can achieve comparable held-out loss to re-training from scratch, while requiring significantly less computation (2.6x). However, the optimal balance between incorporating new data and replaying old data differs as replay is crucial to avoid forgetting on generic web data but less so on specific domains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_02107 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining Li, Jeffrey Armandpour, Mohammadreza Mirzadeh, Iman Mehta, Sachin Shankar, Vaishaal Vemulapalli, Raviteja Bengio, Samy Tuzel, Oncel Farajtabar, Mehrdad Pouransari, Hadi Faghri, Fartash Machine Learning Computation and Language Large Language Models (LLMs) trained on historical web data inevitably become outdated. We investigate evaluation strategies and update methods for LLMs as new data becomes available. We introduce a web-scale dataset for time-continual pretraining of LLMs derived from 114 dumps of Common Crawl (CC) - orders of magnitude larger than previous continual language modeling benchmarks. We also design time-stratified evaluations across both general CC data and specific domains (Wikipedia, StackExchange, and code documentation) to assess how well various continual learning methods adapt to new data while retaining past knowledge. Our findings demonstrate that, on general CC data, autoregressive meta-schedules combined with a fixed-ratio replay of older data can achieve comparable held-out loss to re-training from scratch, while requiring significantly less computation (2.6x). However, the optimal balance between incorporating new data and replaying old data differs as replay is crucial to avoid forgetting on generic web data but less so on specific domains. |
| title | TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2504.02107 |