Influence-driven Curriculum Learning for Pre-training on Limited Data
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908560171991040 |
|---|---|
| author | Schoenegger, Loris Thoma, Lukas Blevins, Terra Roth, Benjamin |
| author_facet | Schoenegger, Loris Thoma, Lukas Blevins, Terra Roth, Benjamin |
| contents | Curriculum learning, a training technique where data is presented to the model in order of example difficulty (e.g., from simpler to more complex documents), has shown limited success for pre-training language models. In this work, we investigate whether curriculum learning becomes competitive if we replace conventional human-centered difficulty metrics with one that more closely corresponds to example difficulty as observed during model training. Specifically, we experiment with sorting training examples by their \textit{training data influence}, a score which estimates the effect of individual training examples on the model's output. Models trained on our curricula are able to outperform ones trained in random order by over 10 percentage points in benchmarks, confirming that curriculum learning is beneficial for language model pre-training, as long as a more model-centric notion of difficulty is adopted. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_15475 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Influence-driven Curriculum Learning for Pre-training on Limited Data Schoenegger, Loris Thoma, Lukas Blevins, Terra Roth, Benjamin Computation and Language Machine Learning I.2.7 Curriculum learning, a training technique where data is presented to the model in order of example difficulty (e.g., from simpler to more complex documents), has shown limited success for pre-training language models. In this work, we investigate whether curriculum learning becomes competitive if we replace conventional human-centered difficulty metrics with one that more closely corresponds to example difficulty as observed during model training. Specifically, we experiment with sorting training examples by their \textit{training data influence}, a score which estimates the effect of individual training examples on the model's output. Models trained on our curricula are able to outperform ones trained in random order by over 10 percentage points in benchmarks, confirming that curriculum learning is beneficial for language model pre-training, as long as a more model-centric notion of difficulty is adopted. |
| title | Influence-driven Curriculum Learning for Pre-training on Limited Data |
| topic | Computation and Language Machine Learning I.2.7 |
| url | https://arxiv.org/abs/2508.15475 |