CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915529583755264 |
|---|---|
| author | Liu, Ziyue Zhang, Ruijie Wang, Zhengyang Yan, Mingsong Yang, Zi Hovland, Paul Nicolae, Bogdan Cappello, Franck Tang, Sui Zhang, Zheng |
| author_facet | Liu, Ziyue Zhang, Ruijie Wang, Zhengyang Yan, Mingsong Yang, Zi Hovland, Paul Nicolae, Bogdan Cappello, Franck Tang, Sui Zhang, Zheng |
| contents | The full-size MLPs and the projection layers in attention introduce tremendous model sizes of large language models (LLMs), consuming extensive computational resources in pre-training. We empirically observe that the activations of pre-trained LLMs exhibit low-rank property. Motivated by such observations, we propose CoLA and its memory-efficient implementation, CoLA-M, to replace these full-size layers with compute-efficient auto-encoders that naturally enforce low-rank activations throughout training. This fundamental architectural change eliminates the activation redundancy and significantly boosts model capacity and training efficiency. Experiments on LLaMA models with 60 million to 7 billion parameters show that CoLA reduces the computing cost by $\bf 2\pmb{\times}$ and improves training throughput by $\bf 1.86\pmb{\times}$ while maintaining full-rank level performance. CoLA-M further squeezes memory cost without sacrificing throughput, offering a pre-training approach with collectively superior parameter, computing, and memory efficiency. The LLMs produced are also $\bf 2\pmb{\times}$ smaller, enabling faster inference with lower memory cost on resource-constrained platforms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_10940 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation Liu, Ziyue Zhang, Ruijie Wang, Zhengyang Yan, Mingsong Yang, Zi Hovland, Paul Nicolae, Bogdan Cappello, Franck Tang, Sui Zhang, Zheng Machine Learning Artificial Intelligence The full-size MLPs and the projection layers in attention introduce tremendous model sizes of large language models (LLMs), consuming extensive computational resources in pre-training. We empirically observe that the activations of pre-trained LLMs exhibit low-rank property. Motivated by such observations, we propose CoLA and its memory-efficient implementation, CoLA-M, to replace these full-size layers with compute-efficient auto-encoders that naturally enforce low-rank activations throughout training. This fundamental architectural change eliminates the activation redundancy and significantly boosts model capacity and training efficiency. Experiments on LLaMA models with 60 million to 7 billion parameters show that CoLA reduces the computing cost by $\bf 2\pmb{\times}$ and improves training throughput by $\bf 1.86\pmb{\times}$ while maintaining full-rank level performance. CoLA-M further squeezes memory cost without sacrificing throughput, offering a pre-training approach with collectively superior parameter, computing, and memory efficiency. The LLMs produced are also $\bf 2\pmb{\times}$ smaller, enabling faster inference with lower memory cost on resource-constrained platforms. |
| title | CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2502.10940 |