Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2410.14268 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909354131718144 |
|---|---|
| author | Zhang, Chen Zhong, Meizhi Wang, Qimeng Lu, Xuantao Ye, Zheyu Lu, Chengqiang Gao, Yan Hu, Yao Chen, Kehai Zhang, Min Song, Dawei |
| author_facet | Zhang, Chen Zhong, Meizhi Wang, Qimeng Lu, Xuantao Ye, Zheyu Lu, Chengqiang Gao, Yan Hu, Yao Chen, Kehai Zhang, Min Song, Dawei |
| contents | Long-context efficiency has recently become a trending topic in serving large language models (LLMs). And mixture of depths (MoD) is proposed as a perfect fit to bring down both latency and memory. In this paper, however, we discover that MoD can barely transform existing LLMs without costly training over an extensive number of tokens. To enable the transformations from any LLMs to MoD ones, we showcase top-k operator in MoD should be promoted to threshold-p operator, and refinement to architecture and data should also be crafted along. All these designs form our method termed MoDification. Through a comprehensive set of experiments covering model scales from 3B to 70B, we exhibit MoDification strikes an excellent balance between efficiency and effectiveness. MoDification can achieve up to ~1.2x speedup in latency and ~1.8x reduction in memory compared to original LLMs especially in long-context applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_14268 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MoDification: Mixture of Depths Made Easy Zhang, Chen Zhong, Meizhi Wang, Qimeng Lu, Xuantao Ye, Zheyu Lu, Chengqiang Gao, Yan Hu, Yao Chen, Kehai Zhang, Min Song, Dawei Computation and Language Machine Learning Long-context efficiency has recently become a trending topic in serving large language models (LLMs). And mixture of depths (MoD) is proposed as a perfect fit to bring down both latency and memory. In this paper, however, we discover that MoD can barely transform existing LLMs without costly training over an extensive number of tokens. To enable the transformations from any LLMs to MoD ones, we showcase top-k operator in MoD should be promoted to threshold-p operator, and refinement to architecture and data should also be crafted along. All these designs form our method termed MoDification. Through a comprehensive set of experiments covering model scales from 3B to 70B, we exhibit MoDification strikes an excellent balance between efficiency and effectiveness. MoDification can achieve up to ~1.2x speedup in latency and ~1.8x reduction in memory compared to original LLMs especially in long-context applications. |
| title | MoDification: Mixture of Depths Made Easy |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2410.14268 |