Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909640110899200 |
|---|---|
| author | Kim, Jeonghoon Lee, Byeongchan Park, Cheonbok Oh, Yeontaek Kim, Beomjun Yoo, Taehwan Shin, Seongjin Han, Dongyoon Shin, Jinwoo Yoo, Kang Min |
| author_facet | Kim, Jeonghoon Lee, Byeongchan Park, Cheonbok Oh, Yeontaek Kim, Beomjun Yoo, Taehwan Shin, Seongjin Han, Dongyoon Shin, Jinwoo Yoo, Kang Min |
| contents | Selecting a layer normalization (LN) strategy that stabilizes training and speeds convergence in Transformers remains difficult, even for today's large language models (LLM). We present a comprehensive analytical foundation for understanding how different LN strategies influence training dynamics in large-scale Transformers. Until recently, Pre-LN and Post-LN have long dominated practices despite their limitations in large-scale training. However, several open-source models have recently begun silently adopting a third strategy without much explanation. This strategy places normalization layer peripherally around sublayers, a design we term Peri-LN. While Peri-LN has demonstrated promising performance, its precise mechanisms and benefits remain almost unexplored. Our in-depth analysis delineates the distinct behaviors of LN strategies, showing how each placement shapes activation variance and gradient propagation. To validate our theoretical insight, we conduct extensive experiments on Transformers up to $3.2$B parameters, showing that Peri-LN consistently achieves more balanced variance growth, steadier gradient flow, and convergence stability. Our results suggest that Peri-LN warrants broader consideration for large-scale Transformer architectures, providing renewed insights into the optimal placement of LN. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_02732 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Peri-LN: Revisiting Normalization Layer in the Transformer Architecture Kim, Jeonghoon Lee, Byeongchan Park, Cheonbok Oh, Yeontaek Kim, Beomjun Yoo, Taehwan Shin, Seongjin Han, Dongyoon Shin, Jinwoo Yoo, Kang Min Machine Learning Artificial Intelligence Computation and Language Selecting a layer normalization (LN) strategy that stabilizes training and speeds convergence in Transformers remains difficult, even for today's large language models (LLM). We present a comprehensive analytical foundation for understanding how different LN strategies influence training dynamics in large-scale Transformers. Until recently, Pre-LN and Post-LN have long dominated practices despite their limitations in large-scale training. However, several open-source models have recently begun silently adopting a third strategy without much explanation. This strategy places normalization layer peripherally around sublayers, a design we term Peri-LN. While Peri-LN has demonstrated promising performance, its precise mechanisms and benefits remain almost unexplored. Our in-depth analysis delineates the distinct behaviors of LN strategies, showing how each placement shapes activation variance and gradient propagation. To validate our theoretical insight, we conduct extensive experiments on Transformers up to $3.2$B parameters, showing that Peri-LN consistently achieves more balanced variance growth, steadier gradient flow, and convergence stability. Our results suggest that Peri-LN warrants broader consideration for large-scale Transformer architectures, providing renewed insights into the optimal placement of LN. |
| title | Peri-LN: Revisiting Normalization Layer in the Transformer Architecture |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2502.02732 |