GLAI: GreenLightningAI for Accelerated Training through Knowledge Decoupling
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911668277084160 |
|---|---|
| author | Mestre, Jose I. Fernández-Hernández, Alberto Pérez-Corral, Cristian Dolz, Manuel F. Duato, Jose Quintana-Ortí, Enrique S. |
| author_facet | Mestre, Jose I. Fernández-Hernández, Alberto Pérez-Corral, Cristian Dolz, Manuel F. Duato, Jose Quintana-Ortí, Enrique S. |
| contents | In this work we introduce GreenLightningAI (GLAI), a new architectural block designed as an alternative to conventional MLPs. The central idea is to separate two types of knowledge that are usually entangled during training: (i) *structural knowledge*, encoded by the stable activation patterns induced by ReLU activations; and (ii) *quantitative knowledge*, carried by the numerical weights and biases. By fixing the structure once stabilized, GLAI reformulates the MLP as a combination of paths, where only the quantitative component is optimized. This reformulation retains the universal approximation capabilities of MLPs, yet achieves a more efficient training process, reducing training time by ~40% on average across the cases examined in this study. Crucially, GLAI is not just another classifier, but a generic block that can replace MLPs wherever they are used, from supervised heads with frozen backbones to projection layers in self-supervised learning or few-shot classifiers. Across diverse experimental setups, GLAI consistently matches or exceeds the accuracy of MLPs with an equivalent number of parameters, while converging faster. Overall, GLAI establishes a new design principle that opens a direction for future integration into large-scale architectures such as Transformers, where MLP blocks dominate the computational footprint. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_00883 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GLAI: GreenLightningAI for Accelerated Training through Knowledge Decoupling Mestre, Jose I. Fernández-Hernández, Alberto Pérez-Corral, Cristian Dolz, Manuel F. Duato, Jose Quintana-Ortí, Enrique S. Machine Learning Artificial Intelligence In this work we introduce GreenLightningAI (GLAI), a new architectural block designed as an alternative to conventional MLPs. The central idea is to separate two types of knowledge that are usually entangled during training: (i) *structural knowledge*, encoded by the stable activation patterns induced by ReLU activations; and (ii) *quantitative knowledge*, carried by the numerical weights and biases. By fixing the structure once stabilized, GLAI reformulates the MLP as a combination of paths, where only the quantitative component is optimized. This reformulation retains the universal approximation capabilities of MLPs, yet achieves a more efficient training process, reducing training time by ~40% on average across the cases examined in this study. Crucially, GLAI is not just another classifier, but a generic block that can replace MLPs wherever they are used, from supervised heads with frozen backbones to projection layers in self-supervised learning or few-shot classifiers. Across diverse experimental setups, GLAI consistently matches or exceeds the accuracy of MLPs with an equivalent number of parameters, while converging faster. Overall, GLAI establishes a new design principle that opens a direction for future integration into large-scale architectures such as Transformers, where MLP blocks dominate the computational footprint. |
| title | GLAI: GreenLightningAI for Accelerated Training through Knowledge Decoupling |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2510.00883 |