DenseMamba: State Space Models with Dense Hidden Connection for Efficient Large Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929264922722304 |
|---|---|
| author | He, Wei Han, Kai Tang, Yehui Wang, Chengcheng Yang, Yujie Guo, Tianyu Wang, Yunhe |
| author_facet | He, Wei Han, Kai Tang, Yehui Wang, Chengcheng Yang, Yujie Guo, Tianyu Wang, Yunhe |
| contents | Large language models (LLMs) face a daunting challenge due to the excessive computational and memory requirements of the commonly used Transformer architecture. While state space model (SSM) is a new type of foundational network architecture offering lower computational complexity, their performance has yet to fully rival that of Transformers. This paper introduces DenseSSM, a novel approach to enhance the flow of hidden information between layers in SSMs. By selectively integrating shallowlayer hidden states into deeper layers, DenseSSM retains fine-grained information crucial for the final output. Dense connections enhanced DenseSSM still maintains the training parallelizability and inference efficiency. The proposed method can be widely applicable to various SSM types like RetNet and Mamba. With similar model size, DenseSSM achieves significant improvements, exemplified by DenseRetNet outperforming the original RetNet with up to 5% accuracy improvement on public benchmarks. code is avalaible at https://github.com/WailordHe/DenseSSM |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_00818 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DenseMamba: State Space Models with Dense Hidden Connection for Efficient Large Language Models He, Wei Han, Kai Tang, Yehui Wang, Chengcheng Yang, Yujie Guo, Tianyu Wang, Yunhe Computation and Language Machine Learning Large language models (LLMs) face a daunting challenge due to the excessive computational and memory requirements of the commonly used Transformer architecture. While state space model (SSM) is a new type of foundational network architecture offering lower computational complexity, their performance has yet to fully rival that of Transformers. This paper introduces DenseSSM, a novel approach to enhance the flow of hidden information between layers in SSMs. By selectively integrating shallowlayer hidden states into deeper layers, DenseSSM retains fine-grained information crucial for the final output. Dense connections enhanced DenseSSM still maintains the training parallelizability and inference efficiency. The proposed method can be widely applicable to various SSM types like RetNet and Mamba. With similar model size, DenseSSM achieves significant improvements, exemplified by DenseRetNet outperforming the original RetNet with up to 5% accuracy improvement on public benchmarks. code is avalaible at https://github.com/WailordHe/DenseSSM |
| title | DenseMamba: State Space Models with Dense Hidden Connection for Efficient Large Language Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2403.00818 |