A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912247595401216 |
|---|---|
| author | Fu, Shi Wang, Yingjie Chen, Yuzhu Tian, Xinmei Tao, Dacheng |
| author_facet | Fu, Shi Wang, Yingjie Chen, Yuzhu Tian, Xinmei Tao, Dacheng |
| contents | High-quality data is essential for training large generative models, yet the vast reservoir of real data available online has become nearly depleted. Consequently, models increasingly generate their own data for further training, forming Self-consuming Training Loops (STLs). However, the empirical results have been strikingly inconsistent: some models degrade or even collapse, while others successfully avoid these failures, leaving a significant gap in theoretical understanding to explain this discrepancy. This paper introduces the intriguing notion of recursive stability and presents the first theoretical generalization analysis, revealing how both model architecture and the proportion between real and synthetic data influence the success of STLs. We further extend this analysis to transformers in in-context learning, showing that even a constant-sized proportion of real data ensures convergence, while also providing insights into optimal synthetic data sizing. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_18865 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops Fu, Shi Wang, Yingjie Chen, Yuzhu Tian, Xinmei Tao, Dacheng Machine Learning Artificial Intelligence High-quality data is essential for training large generative models, yet the vast reservoir of real data available online has become nearly depleted. Consequently, models increasingly generate their own data for further training, forming Self-consuming Training Loops (STLs). However, the empirical results have been strikingly inconsistent: some models degrade or even collapse, while others successfully avoid these failures, leaving a significant gap in theoretical understanding to explain this discrepancy. This paper introduces the intriguing notion of recursive stability and presents the first theoretical generalization analysis, revealing how both model architecture and the proportion between real and synthetic data influence the success of STLs. We further extend this analysis to transformers in in-context learning, showing that even a constant-sized proportion of real data ensures convergence, while also providing insights into optimal synthetic data sizing. |
| title | A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2502.18865 |