Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.19147 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915947196973056 |
|---|---|
| author | Zhao, Weijie Liu, Mingquan Wang, Bolun Wu, Simo Xie, Nuobei Zhu, Rui-Jie Zhou, Peng |
| author_facet | Zhao, Weijie Liu, Mingquan Wang, Bolun Wu, Simo Xie, Nuobei Zhu, Rui-Jie Zhou, Peng |
| contents | Scaling Transformers typically necessitates training larger models from scratch, as standard architectures struggle to expand without discarding learned representations. We identify the primary bottleneck in the attention mechanism's linear projections, which strictly confine feature extraction to fixed-dimensional subspaces, limiting both expressivity and incremental capacity. To address this, we introduce Nexusformer, which replaces linear $Q/K/V$ projections with a Nexus-Rank layer, a three-stage nonlinear mapping driven by dual activations in progressively higher dimensional spaces. This design overcomes the linearity constraint and enables lossless structured growth: new capacity can be injected along two axes via zero-initialized blocks that preserve pretrained knowledge. Experiments on language modeling and reasoning benchmarks demonstrate that Nexusformer matches Tokenformer's perplexity using up to 41.5\% less training compute during progressive scaling (240M to 440M). Furthermore, our analysis of growth dynamics reveals that zero initialization induces a stable convergence trajectory, allowing us to derive a geometric scaling law that accurately predicts performance across expansion scales. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_19147 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Nexusformer: Nonlinear Attention Expansion for Stable and Inheritable Transformer Scaling Zhao, Weijie Liu, Mingquan Wang, Bolun Wu, Simo Xie, Nuobei Zhu, Rui-Jie Zhou, Peng Machine Learning Artificial Intelligence Scaling Transformers typically necessitates training larger models from scratch, as standard architectures struggle to expand without discarding learned representations. We identify the primary bottleneck in the attention mechanism's linear projections, which strictly confine feature extraction to fixed-dimensional subspaces, limiting both expressivity and incremental capacity. To address this, we introduce Nexusformer, which replaces linear $Q/K/V$ projections with a Nexus-Rank layer, a three-stage nonlinear mapping driven by dual activations in progressively higher dimensional spaces. This design overcomes the linearity constraint and enables lossless structured growth: new capacity can be injected along two axes via zero-initialized blocks that preserve pretrained knowledge. Experiments on language modeling and reasoning benchmarks demonstrate that Nexusformer matches Tokenformer's perplexity using up to 41.5\% less training compute during progressive scaling (240M to 440M). Furthermore, our analysis of growth dynamics reveals that zero initialization induces a stable convergence trajectory, allowing us to derive a geometric scaling law that accurately predicts performance across expansion scales. |
| title | Nexusformer: Nonlinear Attention Expansion for Stable and Inheritable Transformer Scaling |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2604.19147 |