Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.18302 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912849487462400 |
|---|---|
| author | Shibata, Keigo Yano, Kazuki Takahashi, Ryosuke Lee, Jaesung Ikeda, Wataru Suzuki, Jun |
| author_facet | Shibata, Keigo Yano, Kazuki Takahashi, Ryosuke Lee, Jaesung Ikeda, Wataru Suzuki, Jun |
| contents | This paper discusses the internal behavior of Transformer language models. Many recent pre-trained models have been reported to exhibit only slight changes in the angular distance between the input and output hidden state vectors in the middle Transformer layers, despite a disproportionately large ``jump'' in the angular distance occurring in or around the final Transformer layer. To characterize this, we first introduce a quantitative metric for the jump strength around the final layer, and then demonstrate its prevalence across many open-weight models, as well as its amplification throughout pre-training. Assuming such jumps indicate an undesirable property, we propose the jump-suppressing regularizer (JREG) which penalizes this jump during pre-training, thereby encouraging more balanced capability usage across the middle layers. Empirical evaluations of three model sizes of Llama-based models, trained with the proposed JREG method, reveal improved task performance compared to the baseline without altering the model architecture. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_18302 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Suppressing Final Layer Hidden State Jumps in Transformer Pretraining Shibata, Keigo Yano, Kazuki Takahashi, Ryosuke Lee, Jaesung Ikeda, Wataru Suzuki, Jun Computation and Language This paper discusses the internal behavior of Transformer language models. Many recent pre-trained models have been reported to exhibit only slight changes in the angular distance between the input and output hidden state vectors in the middle Transformer layers, despite a disproportionately large ``jump'' in the angular distance occurring in or around the final Transformer layer. To characterize this, we first introduce a quantitative metric for the jump strength around the final layer, and then demonstrate its prevalence across many open-weight models, as well as its amplification throughout pre-training. Assuming such jumps indicate an undesirable property, we propose the jump-suppressing regularizer (JREG) which penalizes this jump during pre-training, thereby encouraging more balanced capability usage across the middle layers. Empirical evaluations of three model sizes of Llama-based models, trained with the proposed JREG method, reveal improved task performance compared to the baseline without altering the model architecture. |
| title | Suppressing Final Layer Hidden State Jumps in Transformer Pretraining |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.18302 |