Efficient Pre-Training of LLMs through Truncated SVD Layers
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917540509253632 |
|---|---|
| author | Kamali, Kaivan Schweighofer, Kajetan Shahrzad, Hormoz Francon, Olivier Hodjat, Babak Miikkulainen, Risto |
| author_facet | Kamali, Kaivan Schweighofer, Kajetan Shahrzad, Hormoz Francon, Olivier Hodjat, Babak Miikkulainen, Risto |
| contents | The massive scaling of Large Language Models (LLMs) has made pretraining increasingly cost-prohibitive. While low-rank representation and orthonormal weight matrices could in principle reduce parameter counts and computational overhead, most existing methods rely on static rank selection and do not enforce weight orthonormality due to high computational cost. This paper introduces TSVD, a framework that maintains low rank and strict orthonormality throughout the training process. It utilizes a spectral energy-based heuristic for adaptive rank selection, and a caching mechanisms to maintain orthonormality. Theoretical analysis justifies the advantage of the approach in pretraining dynamics and experiments across various model scales demonstrate that it is effective empirically. TSVD matches or exceeds the performance of full-parameter baselines while significantly reducing compute requirements. The approach thus offers a well-founded, practical, and scalable path toward efficient high-performance LLM pretraining. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_28573 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Efficient Pre-Training of LLMs through Truncated SVD Layers Kamali, Kaivan Schweighofer, Kajetan Shahrzad, Hormoz Francon, Olivier Hodjat, Babak Miikkulainen, Risto Machine Learning Artificial Intelligence The massive scaling of Large Language Models (LLMs) has made pretraining increasingly cost-prohibitive. While low-rank representation and orthonormal weight matrices could in principle reduce parameter counts and computational overhead, most existing methods rely on static rank selection and do not enforce weight orthonormality due to high computational cost. This paper introduces TSVD, a framework that maintains low rank and strict orthonormality throughout the training process. It utilizes a spectral energy-based heuristic for adaptive rank selection, and a caching mechanisms to maintain orthonormality. Theoretical analysis justifies the advantage of the approach in pretraining dynamics and experiments across various model scales demonstrate that it is effective empirically. TSVD matches or exceeds the performance of full-parameter baselines while significantly reducing compute requirements. The approach thus offers a well-founded, practical, and scalable path toward efficient high-performance LLM pretraining. |
| title | Efficient Pre-Training of LLMs through Truncated SVD Layers |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2605.28573 |