Rethinking Data Mixing from the Perspective of Large Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914461285089280 |
|---|---|
| author | Xu, Yuanjian Sun, Tianze Xu, Changwei Zhao, XinLong Hao, Jianing Chen, Ran Liu, Yang Xu, Ruijie Chen, Stephen Zhang, Guang |
| author_facet | Xu, Yuanjian Sun, Tianze Xu, Changwei Zhao, XinLong Hao, Jianing Chen, Ran Liu, Yang Xu, Ruijie Chen, Stephen Zhang, Guang |
| contents | Data mixing strategy is essential for large language model (LLM) training. Empirical evidence shows that inappropriate strategies can significantly reduce generalization. Although recent methods have improved empirical performance, several fundamental questions remain open: what constitutes a domain, whether human and model perceptions of domains are aligned, and how domain weighting influences generalization. We address these questions by establishing formal connections between gradient dynamics and domain distributions, offering a theoretical framework that clarifies the role of domains in training dynamics. Building on this analysis, we introduce DoGraph, a reweighting framework that formulates data scheduling as a graph-constrained optimization problem. Extensive experiments on GPT-2 models of varying scales demonstrate that DoGraph consistently achieves competitive performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_07963 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Rethinking Data Mixing from the Perspective of Large Language Models Xu, Yuanjian Sun, Tianze Xu, Changwei Zhao, XinLong Hao, Jianing Chen, Ran Liu, Yang Xu, Ruijie Chen, Stephen Zhang, Guang Computation and Language Artificial Intelligence Machine Learning Data mixing strategy is essential for large language model (LLM) training. Empirical evidence shows that inappropriate strategies can significantly reduce generalization. Although recent methods have improved empirical performance, several fundamental questions remain open: what constitutes a domain, whether human and model perceptions of domains are aligned, and how domain weighting influences generalization. We address these questions by establishing formal connections between gradient dynamics and domain distributions, offering a theoretical framework that clarifies the role of domains in training dynamics. Building on this analysis, we introduce DoGraph, a reweighting framework that formulates data scheduling as a graph-constrained optimization problem. Extensive experiments on GPT-2 models of varying scales demonstrate that DoGraph consistently achieves competitive performance. |
| title | Rethinking Data Mixing from the Perspective of Large Language Models |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2604.07963 |