Finedeep: Mitigating Sparse Activation in Dense LLMs via Multi-Layer Fine-Grained Experts
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909500160606208 |
|---|---|
| author | Pan, Leiyu Su, Zhenpeng Lv, Minxuan Xiong, Yizhe Zhang, Xiangwen Lin, Zijia Chen, Hui Han, Jungong Ding, Guiguang Luo, Cheng Zhang, Di Gai, Kun Xiong, Deyi |
| author_facet | Pan, Leiyu Su, Zhenpeng Lv, Minxuan Xiong, Yizhe Zhang, Xiangwen Lin, Zijia Chen, Hui Han, Jungong Ding, Guiguang Luo, Cheng Zhang, Di Gai, Kun Xiong, Deyi |
| contents | Large language models have demonstrated exceptional performance across a wide range of tasks. However, dense models usually suffer from sparse activation, where many activation values tend towards zero (i.e., being inactivated). We argue that this could restrict the efficient exploration of model representation space. To mitigate this issue, we propose Finedeep, a deep-layered fine-grained expert architecture for dense models. Our framework partitions the feed-forward neural network layers of traditional dense models into small experts, arranges them across multiple sub-layers. A novel routing mechanism is proposed to determine each expert's contribution. We conduct extensive experiments across various model sizes, demonstrating that our approach significantly outperforms traditional dense architectures in terms of perplexity and benchmark performance while maintaining a comparable number of parameters and floating-point operations. Moreover, we find that Finedeep achieves optimal results when balancing depth and width, specifically by adjusting the number of expert sub-layers and the number of experts per sub-layer. Empirical results confirm that Finedeep effectively alleviates sparse activation and efficiently utilizes representation capacity in dense models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_12928 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Finedeep: Mitigating Sparse Activation in Dense LLMs via Multi-Layer Fine-Grained Experts Pan, Leiyu Su, Zhenpeng Lv, Minxuan Xiong, Yizhe Zhang, Xiangwen Lin, Zijia Chen, Hui Han, Jungong Ding, Guiguang Luo, Cheng Zhang, Di Gai, Kun Xiong, Deyi Computation and Language Large language models have demonstrated exceptional performance across a wide range of tasks. However, dense models usually suffer from sparse activation, where many activation values tend towards zero (i.e., being inactivated). We argue that this could restrict the efficient exploration of model representation space. To mitigate this issue, we propose Finedeep, a deep-layered fine-grained expert architecture for dense models. Our framework partitions the feed-forward neural network layers of traditional dense models into small experts, arranges them across multiple sub-layers. A novel routing mechanism is proposed to determine each expert's contribution. We conduct extensive experiments across various model sizes, demonstrating that our approach significantly outperforms traditional dense architectures in terms of perplexity and benchmark performance while maintaining a comparable number of parameters and floating-point operations. Moreover, we find that Finedeep achieves optimal results when balancing depth and width, specifically by adjusting the number of expert sub-layers and the number of experts per sub-layer. Empirical results confirm that Finedeep effectively alleviates sparse activation and efficiently utilizes representation capacity in dense models. |
| title | Finedeep: Mitigating Sparse Activation in Dense LLMs via Multi-Layer Fine-Grained Experts |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.12928 |