Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2506.21910 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910014966333440 |
|---|---|
| author | Chang, Ernie Li, Yang Huber, Patrick Vogeti, Vish Kant, David Shi, Yangyang Chandra, Vikas |
| author_facet | Chang, Ernie Li, Yang Huber, Patrick Vogeti, Vish Kant, David Shi, Yangyang Chandra, Vikas |
| contents | In language model training, it is desirable to equip models with capabilities from various tasks. However, it is not clear how to directly obtain the right data mixtures for these capabilities as the relationship between data and tasks is difficult to be modeled. In this work, we observe that checkpoint models exhibit emerging capabilities at different points in the training trajectory. Often, the training process saves checkpoints as artifacts that are under-utilized as a source of in-training data signals. We identify these artifact models based on their respective capabilities on the benchmarks and leverage them as data mixers by using their aggregated first-order influence approximation over source data. We demonstrated on eight reasoning benchmarks that the proposed framework shows significant improvements in the pretraining setting, with performance improvements of up to 1.93%. Overall, this shows the potential of checkpoint models to enhance data quality and optimize data mixtures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_21910 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AutoMixer: Checkpoint Artifacts as Automatic Data Mixers Chang, Ernie Li, Yang Huber, Patrick Vogeti, Vish Kant, David Shi, Yangyang Chandra, Vikas Computation and Language In language model training, it is desirable to equip models with capabilities from various tasks. However, it is not clear how to directly obtain the right data mixtures for these capabilities as the relationship between data and tasks is difficult to be modeled. In this work, we observe that checkpoint models exhibit emerging capabilities at different points in the training trajectory. Often, the training process saves checkpoints as artifacts that are under-utilized as a source of in-training data signals. We identify these artifact models based on their respective capabilities on the benchmarks and leverage them as data mixers by using their aggregated first-order influence approximation over source data. We demonstrated on eight reasoning benchmarks that the proposed framework shows significant improvements in the pretraining setting, with performance improvements of up to 1.93%. Overall, this shows the potential of checkpoint models to enhance data quality and optimize data mixtures. |
| title | AutoMixer: Checkpoint Artifacts as Automatic Data Mixers |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.21910 |