Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.14125 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912280605622272 |
|---|---|
| author | Zhu, Defa Huang, Hongzhi Zhou, Jundong Huang, Zihao Zeng, Yutao Wu, Banggu Min, Qiyang Zhou, Xun |
| author_facet | Zhu, Defa Huang, Hongzhi Zhou, Jundong Huang, Zihao Zeng, Yutao Wu, Banggu Min, Qiyang Zhou, Xun |
| contents | Residual connections are central to modern deep learning architectures, enabling the training of very deep networks by mitigating gradient vanishing. Hyper-Connections recently generalized residual connections by introducing multiple connection strengths at different depths, thereby addressing the seesaw effect between gradient vanishing and representation collapse. However, Hyper-Connections increase memory access costs by expanding the width of hidden states. In this paper, we propose Frac-Connections, a novel approach that divides hidden states into multiple parts rather than expanding their width. Frac-Connections retain partial benefits of Hyper-Connections while reducing memory consumption. To validate their effectiveness, we conduct large-scale experiments on language tasks, with the largest being a 7B MoE model trained on up to 3T tokens, demonstrating that Frac-Connections significantly outperform residual connections. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_14125 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Frac-Connections: Fractional Extension of Hyper-Connections Zhu, Defa Huang, Hongzhi Zhou, Jundong Huang, Zihao Zeng, Yutao Wu, Banggu Min, Qiyang Zhou, Xun Machine Learning Artificial Intelligence Computation and Language Residual connections are central to modern deep learning architectures, enabling the training of very deep networks by mitigating gradient vanishing. Hyper-Connections recently generalized residual connections by introducing multiple connection strengths at different depths, thereby addressing the seesaw effect between gradient vanishing and representation collapse. However, Hyper-Connections increase memory access costs by expanding the width of hidden states. In this paper, we propose Frac-Connections, a novel approach that divides hidden states into multiple parts rather than expanding their width. Frac-Connections retain partial benefits of Hyper-Connections while reducing memory consumption. To validate their effectiveness, we conduct large-scale experiments on language tasks, with the largest being a 7B MoE model trained on up to 3T tokens, demonstrating that Frac-Connections significantly outperform residual connections. |
| title | Frac-Connections: Fractional Extension of Hyper-Connections |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2503.14125 |