GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914292020805632 |
|---|---|
| author | Zheng, Chuanyang Sun, Jiankai Gao, Yihang Wang, Chi Wang, Yuehao Xiong, Jing Ren, Liliang Peng, Bo Wang, Qingmei Shang, Xiaoran Schwager, Mac Schneider, Anderson Nevmyvaka, Yuriy Liu, Xiaodong |
| author_facet | Zheng, Chuanyang Sun, Jiankai Gao, Yihang Wang, Chi Wang, Yuehao Xiong, Jing Ren, Liliang Peng, Bo Wang, Qingmei Shang, Xiaoran Schwager, Mac Schneider, Anderson Nevmyvaka, Yuriy Liu, Xiaodong |
| contents | The placement of normalization layers, specifically Pre-Norm and Post-Norm, remains an open question in Transformer architecture design. In this work, we rethink these approaches through the lens of manifold optimization, interpreting the outputs of the Feed-Forward Network (FFN) and attention layers as update directions in optimization. Building on this perspective, we introduce GeoNorm, a novel method that replaces standard normalization with geodesic updates on the manifold. Furthermore, analogous to learning rate schedules, we propose a layer-wise update decay for the FFN and attention components. Comprehensive experiments demonstrate that GeoNorm consistently outperforms existing normalization methods in Transformer models. Crucially, GeoNorm can be seamlessly integrated into standard Transformer architectures, achieving performance improvements with negligible additional computational cost. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_22095 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization Zheng, Chuanyang Sun, Jiankai Gao, Yihang Wang, Chi Wang, Yuehao Xiong, Jing Ren, Liliang Peng, Bo Wang, Qingmei Shang, Xiaoran Schwager, Mac Schneider, Anderson Nevmyvaka, Yuriy Liu, Xiaodong Machine Learning Computation and Language The placement of normalization layers, specifically Pre-Norm and Post-Norm, remains an open question in Transformer architecture design. In this work, we rethink these approaches through the lens of manifold optimization, interpreting the outputs of the Feed-Forward Network (FFN) and attention layers as update directions in optimization. Building on this perspective, we introduce GeoNorm, a novel method that replaces standard normalization with geodesic updates on the manifold. Furthermore, analogous to learning rate schedules, we propose a layer-wise update decay for the FFN and attention components. Comprehensive experiments demonstrate that GeoNorm consistently outperforms existing normalization methods in Transformer models. Crucially, GeoNorm can be seamlessly integrated into standard Transformer architectures, achieving performance improvements with negligible additional computational cost. |
| title | GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2601.22095 |