Analyzing the Training Dynamics of Image Restoration Transformers: A Revisit to Layer Normalization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912914056675328 |
|---|---|
| author | Lee, MinKyu Hyun, Sangeek Jun, Woojin Kim, Hyunjun Chung, Jiwoo Heo, Jae-Pil |
| author_facet | Lee, MinKyu Hyun, Sangeek Jun, Woojin Kim, Hyunjun Chung, Jiwoo Heo, Jae-Pil |
| contents | This work analyzes the training dynamics of Image Restoration (IR) Transformers and uncovers a critical yet overlooked issue: conventional LayerNorm (LN) drives feature magnitudes to diverge to a million scale and collapses channel-wise entropy. We analyze this in the perspective of networks attempting to bypass LN's constraints that conflict with IR tasks. Accordingly, we address two misalignments between LN and IR: 1) per-token normalization disrupts spatial correlations, and 2) input-independent scaling discards input-specific statistics. To address this, we propose Image Restoration Transformer Tailored Layer Normalization i-LN, a simple drop-in replacement that normalizes features holistically and adaptively rescales them per input. We provide theoretical insights and empirical evidence that this simple design effectively leads to both improved training dynamics and thereby improved performance, validated by extensive experiments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_06629 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Analyzing the Training Dynamics of Image Restoration Transformers: A Revisit to Layer Normalization Lee, MinKyu Hyun, Sangeek Jun, Woojin Kim, Hyunjun Chung, Jiwoo Heo, Jae-Pil Computer Vision and Pattern Recognition This work analyzes the training dynamics of Image Restoration (IR) Transformers and uncovers a critical yet overlooked issue: conventional LayerNorm (LN) drives feature magnitudes to diverge to a million scale and collapses channel-wise entropy. We analyze this in the perspective of networks attempting to bypass LN's constraints that conflict with IR tasks. Accordingly, we address two misalignments between LN and IR: 1) per-token normalization disrupts spatial correlations, and 2) input-independent scaling discards input-specific statistics. To address this, we propose Image Restoration Transformer Tailored Layer Normalization i-LN, a simple drop-in replacement that normalizes features holistically and adaptively rescales them per input. We provide theoretical insights and empirical evidence that this simple design effectively leads to both improved training dynamics and thereby improved performance, validated by extensive experiments. |
| title | Analyzing the Training Dynamics of Image Restoration Transformers: A Revisit to Layer Normalization |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2504.06629 |