mHC: Manifold-Constrained Hyper-Connections
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914233719980032 |
|---|---|
| author | Xie, Zhenda Wei, Yixuan Cao, Huanqi Zhao, Chenggang Deng, Chengqi Li, Jiashi Dai, Damai Gao, Huazuo Chang, Jiang Yu, Kuai Zhao, Liang Zhou, Shangyan Xu, Zhean Zhang, Zhengyan Zeng, Wangding Hu, Shengding Wang, Yuqing Yuan, Jingyang Wang, Lean Liang, Wenfeng |
| author_facet | Xie, Zhenda Wei, Yixuan Cao, Huanqi Zhao, Chenggang Deng, Chengqi Li, Jiashi Dai, Damai Gao, Huazuo Chang, Jiang Yu, Kuai Zhao, Liang Zhou, Shangyan Xu, Zhean Zhang, Zhengyan Zeng, Wangding Hu, Shengding Wang, Yuqing Yuan, Jingyang Wang, Lean Liang, Wenfeng |
| contents | Recently, studies exemplified by Hyper-Connections (HC) have extended the ubiquitous residual connection paradigm established over the past decade by expanding the residual stream width and diversifying connectivity patterns. While yielding substantial performance gains, this diversification fundamentally compromises the identity mapping property intrinsic to the residual connection, which causes severe training instability and restricted scalability, and additionally incurs notable memory access overhead. To address these challenges, we propose Manifold-Constrained Hyper-Connections (mHC), a general framework that projects the residual connection space of HC onto a specific manifold to restore the identity mapping property, while incorporating rigorous infrastructure optimization to ensure efficiency. Empirical experiments demonstrate that mHC is effective for training at scale, offering tangible performance improvements and superior scalability. We anticipate that mHC, as a flexible and practical extension of HC, will contribute to a deeper understanding of topological architecture design and suggest promising directions for the evolution of foundational models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_24880 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | mHC: Manifold-Constrained Hyper-Connections Xie, Zhenda Wei, Yixuan Cao, Huanqi Zhao, Chenggang Deng, Chengqi Li, Jiashi Dai, Damai Gao, Huazuo Chang, Jiang Yu, Kuai Zhao, Liang Zhou, Shangyan Xu, Zhean Zhang, Zhengyan Zeng, Wangding Hu, Shengding Wang, Yuqing Yuan, Jingyang Wang, Lean Liang, Wenfeng Computation and Language Artificial Intelligence Machine Learning Recently, studies exemplified by Hyper-Connections (HC) have extended the ubiquitous residual connection paradigm established over the past decade by expanding the residual stream width and diversifying connectivity patterns. While yielding substantial performance gains, this diversification fundamentally compromises the identity mapping property intrinsic to the residual connection, which causes severe training instability and restricted scalability, and additionally incurs notable memory access overhead. To address these challenges, we propose Manifold-Constrained Hyper-Connections (mHC), a general framework that projects the residual connection space of HC onto a specific manifold to restore the identity mapping property, while incorporating rigorous infrastructure optimization to ensure efficiency. Empirical experiments demonstrate that mHC is effective for training at scale, offering tangible performance improvements and superior scalability. We anticipate that mHC, as a flexible and practical extension of HC, will contribute to a deeper understanding of topological architecture design and suggest promising directions for the evolution of foundational models. |
| title | mHC: Manifold-Constrained Hyper-Connections |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2512.24880 |