Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.12364 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913680726163456 |
|---|---|
| author | Huang, Zihao Min, Qiyang Huang, Hongzhi Zhu, Defa Zeng, Yutao Guo, Ran Zhou, Xun |
| author_facet | Huang, Zihao Min, Qiyang Huang, Hongzhi Zhu, Defa Zeng, Yutao Guo, Ran Zhou, Xun |
| contents | It is widely acknowledged that the performance of Transformer models is logarithmically related to their number of parameters and computational complexity. While approaches like Mixture of Experts (MoE) decouple parameter count from computational complexity, they still face challenges in inference due to high memory access costs. This work introduces UltraMem, incorporating large-scale, ultra-sparse memory layer to address these limitations. Our approach significantly reduces inference latency while maintaining model performance. We also investigate the scaling laws of this new architecture, demonstrating that it not only exhibits favorable scaling properties but outperforms MoE. In experiments, the largest UltraMem we train has 20 million memory slots. The results show that our method achieves state-of-the-art inference speed and model performance within a given computational budget, paving the way for billions of slots or experts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_12364 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Ultra-Sparse Memory Network Huang, Zihao Min, Qiyang Huang, Hongzhi Zhu, Defa Zeng, Yutao Guo, Ran Zhou, Xun Machine Learning It is widely acknowledged that the performance of Transformer models is logarithmically related to their number of parameters and computational complexity. While approaches like Mixture of Experts (MoE) decouple parameter count from computational complexity, they still face challenges in inference due to high memory access costs. This work introduces UltraMem, incorporating large-scale, ultra-sparse memory layer to address these limitations. Our approach significantly reduces inference latency while maintaining model performance. We also investigate the scaling laws of this new architecture, demonstrating that it not only exhibits favorable scaling properties but outperforms MoE. In experiments, the largest UltraMem we train has 20 million memory slots. The results show that our method achieves state-of-the-art inference speed and model performance within a given computational budget, paving the way for billions of slots or experts. |
| title | Ultra-Sparse Memory Network |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2411.12364 |