Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.10504 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918494554030080 |
|---|---|
| author | Zhu, Jinchang Li, Jindong Hao, Yuwen Zou, Chengyu Fu, Rong Yang, Menglin |
| author_facet | Zhu, Jinchang Li, Jindong Hao, Yuwen Zou, Chengyu Fu, Rong Yang, Menglin |
| contents | A causal-decoder block is hierarchical: lower layers build the residual basis that upper layers attend over. We identify a failure mode in GPT pretraining: upper layers commit to sharp attention patterns before lower-layer features stabilize. We call this premature upper-layer attention specialization. Temporarily slowing only upper-layer Q/K projections during early training improves final perplexity and downstream accuracy without altering other parameters; it prevents upper attention from collapsing onto an immature residual basis. In LLaMA-style blocks, the same intervention is nearly unnecessary. Through ablations, we isolate multiplicative gated FFNs (not RMSNorm or bias removal) as the component that suppresses the upstream residual writes driving the failure. A pathwise analysis unifies both findings: the learning-rate intervention reduces a step-size factor, while gated FFNs reduce a residual-energy factor on the same growth pathway. Our results identify upper-layer Q/K timing as a concrete interaction point between decoder architecture and optimization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_10504 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining Zhu, Jinchang Li, Jindong Hao, Yuwen Zou, Chengyu Fu, Rong Yang, Menglin Computation and Language A causal-decoder block is hierarchical: lower layers build the residual basis that upper layers attend over. We identify a failure mode in GPT pretraining: upper layers commit to sharp attention patterns before lower-layer features stabilize. We call this premature upper-layer attention specialization. Temporarily slowing only upper-layer Q/K projections during early training improves final perplexity and downstream accuracy without altering other parameters; it prevents upper attention from collapsing onto an immature residual basis. In LLaMA-style blocks, the same intervention is nearly unnecessary. Through ablations, we isolate multiplicative gated FFNs (not RMSNorm or bias removal) as the component that suppresses the upstream residual writes driving the failure. A pathwise analysis unifies both findings: the learning-rate intervention reduces a step-size factor, while gated FFNs reduce a residual-energy factor on the same growth pathway. Our results identify upper-layer Q/K timing as a concrete interaction point between decoder architecture and optimization. |
| title | Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.10504 |