Reparameterized LLM Training via Orthogonal Equivalence Transformation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908704645840896 |
|---|---|
| author | Qiu, Zeju Buchholz, Simon Xiao, Tim Z. Dax, Maximilian Schölkopf, Bernhard Liu, Weiyang |
| author_facet | Qiu, Zeju Buchholz, Simon Xiao, Tim Z. Dax, Maximilian Schölkopf, Bernhard Liu, Weiyang |
| contents | While large language models (LLMs) are driving the rapid advancement of artificial intelligence, effectively and reliably training these large models remains one of the field's most significant challenges. To address this challenge, we propose POET, a novel reParameterized training algorithm that uses Orthogonal Equivalence Transformation to optimize neurons. Specifically, POET reparameterizes each neuron with two learnable orthogonal matrices and a fixed random weight matrix. Because of its provable preservation of spectral properties of weight matrices, POET can stably optimize the objective function with improved generalization. We further develop efficient approximations that make POET flexible and scalable for training large-scale neural networks. Extensive experiments validate the effectiveness and scalability of POET in training LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_08001 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Reparameterized LLM Training via Orthogonal Equivalence Transformation Qiu, Zeju Buchholz, Simon Xiao, Tim Z. Dax, Maximilian Schölkopf, Bernhard Liu, Weiyang Machine Learning Artificial Intelligence Computation and Language While large language models (LLMs) are driving the rapid advancement of artificial intelligence, effectively and reliably training these large models remains one of the field's most significant challenges. To address this challenge, we propose POET, a novel reParameterized training algorithm that uses Orthogonal Equivalence Transformation to optimize neurons. Specifically, POET reparameterizes each neuron with two learnable orthogonal matrices and a fixed random weight matrix. Because of its provable preservation of spectral properties of weight matrices, POET can stably optimize the objective function with improved generalization. We further develop efficient approximations that make POET flexible and scalable for training large-scale neural networks. Extensive experiments validate the effectiveness and scalability of POET in training LLMs. |
| title | Reparameterized LLM Training via Orthogonal Equivalence Transformation |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2506.08001 |