nGPT: Normalized Transformer with Representation Learning on the Hypersphere
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917996610453504 |
|---|---|
| author | Loshchilov, Ilya Hsieh, Cheng-Ping Sun, Simeng Ginsburg, Boris |
| author_facet | Loshchilov, Ilya Hsieh, Cheng-Ping Sun, Simeng Ginsburg, Boris |
| contents | We propose a novel neural network architecture, the normalized Transformer (nGPT) with representation learning on the hypersphere. In nGPT, all vectors forming the embeddings, MLP, attention matrices and hidden states are unit norm normalized. The input stream of tokens travels on the surface of a hypersphere, with each layer contributing a displacement towards the target output predictions. These displacements are defined by the MLP and attention blocks, whose vector components also reside on the same hypersphere. Experiments show that nGPT learns much faster, reducing the number of training steps required to achieve the same accuracy by a factor of 4 to 20, depending on the sequence length. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_01131 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | nGPT: Normalized Transformer with Representation Learning on the Hypersphere Loshchilov, Ilya Hsieh, Cheng-Ping Sun, Simeng Ginsburg, Boris Machine Learning Artificial Intelligence We propose a novel neural network architecture, the normalized Transformer (nGPT) with representation learning on the hypersphere. In nGPT, all vectors forming the embeddings, MLP, attention matrices and hidden states are unit norm normalized. The input stream of tokens travels on the surface of a hypersphere, with each layer contributing a displacement towards the target output predictions. These displacements are defined by the MLP and attention blocks, whose vector components also reside on the same hypersphere. Experiments show that nGPT learns much faster, reducing the number of training steps required to achieve the same accuracy by a factor of 4 to 20, depending on the sequence length. |
| title | nGPT: Normalized Transformer with Representation Learning on the Hypersphere |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2410.01131 |