Enhanced QKNorm normalization for neural transformers with the Lp norm
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918323730513920 |
|---|---|
| author | Lopez-Rubio, Ezequiel Montes-Perez, Javier Palomo, Esteban Jose |
| author_facet | Lopez-Rubio, Ezequiel Montes-Perez, Javier Palomo, Esteban Jose |
| contents | The normalization of query and key vectors is an essential part of the Transformer architecture. It ensures that learning is stable regardless of the scale of these vectors. Some normalization approaches are available. In this preliminary work, a generalization of the QKNorm normalization scheme is proposed. The approach is based on the Lp norm, allowing non-Euclidean norms to be employed. Experimental results demonstrate the suitability of the method for a simple problem. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_05006 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Enhanced QKNorm normalization for neural transformers with the Lp norm Lopez-Rubio, Ezequiel Montes-Perez, Javier Palomo, Esteban Jose Machine Learning Artificial Intelligence Computation and Language 68T07 The normalization of query and key vectors is an essential part of the Transformer architecture. It ensures that learning is stable regardless of the scale of these vectors. Some normalization approaches are available. In this preliminary work, a generalization of the QKNorm normalization scheme is proposed. The approach is based on the Lp norm, allowing non-Euclidean norms to be employed. Experimental results demonstrate the suitability of the method for a simple problem. |
| title | Enhanced QKNorm normalization for neural transformers with the Lp norm |
| topic | Machine Learning Artificial Intelligence Computation and Language 68T07 |
| url | https://arxiv.org/abs/2602.05006 |