SIGMA: Scalable Spectral Insights for LLM Model Collapse
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917356021743616 |
|---|---|
| author | Gu, Yi Pang, Lingyou Ye, Xiangkun Wang, Tianyu Lin, Jianyu Priebe, Carey E. Aue, Alexander |
| author_facet | Gu, Yi Pang, Lingyou Ye, Xiangkun Wang, Tianyu Lin, Jianyu Priebe, Carey E. Aue, Alexander |
| contents | The rapid adoption of synthetic data for training Large Language Models (LLMs) has introduced the technical challenge of "model collapse"-a degenerative process where recursive training on model-generated content leads to a contraction of distributional variance and representational quality. While the phenomenology of collapse is increasingly evident, rigorous methods to quantify and predict its onset in high-dimensional spaces remain elusive. In this paper, we introduce SIGMA (Spectral Inequalities for Gram Matrix Analysis), a unified framework that benchmarks model collapse through the spectral lens of the embedding Gram matrix. By deriving and utilizing deterministic and stochastic bounds on the matrix's spectrum, SIGMA provides a mathematically grounded metric to track the contraction of the representation space. Crucially, our stochastic formulation enables scalable estimation of these bounds, making the framework applicable to large-scale foundation models where full eigendecomposition is intractable. We demonstrate that SIGMA effectively captures the transition towards degenerate states, offering both theoretical insights into the mechanics of collapse and a practical, scalable tool for monitoring the health of recursive training pipelines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_03385 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SIGMA: Scalable Spectral Insights for LLM Model Collapse Gu, Yi Pang, Lingyou Ye, Xiangkun Wang, Tianyu Lin, Jianyu Priebe, Carey E. Aue, Alexander Machine Learning Probability 68T50(Primary), 60F05(Secondary) I.2.7; G.3 The rapid adoption of synthetic data for training Large Language Models (LLMs) has introduced the technical challenge of "model collapse"-a degenerative process where recursive training on model-generated content leads to a contraction of distributional variance and representational quality. While the phenomenology of collapse is increasingly evident, rigorous methods to quantify and predict its onset in high-dimensional spaces remain elusive. In this paper, we introduce SIGMA (Spectral Inequalities for Gram Matrix Analysis), a unified framework that benchmarks model collapse through the spectral lens of the embedding Gram matrix. By deriving and utilizing deterministic and stochastic bounds on the matrix's spectrum, SIGMA provides a mathematically grounded metric to track the contraction of the representation space. Crucially, our stochastic formulation enables scalable estimation of these bounds, making the framework applicable to large-scale foundation models where full eigendecomposition is intractable. We demonstrate that SIGMA effectively captures the transition towards degenerate states, offering both theoretical insights into the mechanics of collapse and a practical, scalable tool for monitoring the health of recursive training pipelines. |
| title | SIGMA: Scalable Spectral Insights for LLM Model Collapse |
| topic | Machine Learning Probability 68T50(Primary), 60F05(Secondary) I.2.7; G.3 |
| url | https://arxiv.org/abs/2601.03385 |