Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2404.06709 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917712698015744 |
|---|---|
| author | Zou, Longwei Wang, Qingyang Zhao, Han Kong, Jiangang Yang, Yi Deng, Yangdong |
| author_facet | Zou, Longwei Wang, Qingyang Zhao, Han Kong, Jiangang Yang, Yi Deng, Yangdong |
| contents | The fast-growing large scale language models are delivering unprecedented performance on almost all natural language processing tasks. However, the effectiveness of large language models are reliant on an exponentially increasing number of parameters. The overwhelming computation complexity incurs a high inference latency that negatively affects user experience. Existing methods to improve inference efficiency, such as tensor parallelism and quantization, target to reduce per-layer computing latency, yet overlook the cumulative latency due to the number of layers. Recent works on reducing the cumulative latency through layer removing, however, lead to significant performance drop. Motivated by the similarity of inputs among adjacent layers, we propose to identify quasi-independent layers, which can be concurrently computed to significantly decrease inference latency. We also introduce a bypassing technique to mitigate the effect of information loss. Empirical experiments of the proposed approach on the LLaMA models confirm that Concurrent Computation of Quasi-Independent Layers (CQIL) can reduce latency by up to 48.3% on LLaMA-33B, while maintaining a close level of performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_06709 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CQIL: Inference Latency Optimization with Concurrent Computation of Quasi-Independent Layers Zou, Longwei Wang, Qingyang Zhao, Han Kong, Jiangang Yang, Yi Deng, Yangdong Computation and Language The fast-growing large scale language models are delivering unprecedented performance on almost all natural language processing tasks. However, the effectiveness of large language models are reliant on an exponentially increasing number of parameters. The overwhelming computation complexity incurs a high inference latency that negatively affects user experience. Existing methods to improve inference efficiency, such as tensor parallelism and quantization, target to reduce per-layer computing latency, yet overlook the cumulative latency due to the number of layers. Recent works on reducing the cumulative latency through layer removing, however, lead to significant performance drop. Motivated by the similarity of inputs among adjacent layers, we propose to identify quasi-independent layers, which can be concurrently computed to significantly decrease inference latency. We also introduce a bypassing technique to mitigate the effect of information loss. Empirical experiments of the proposed approach on the LLaMA models confirm that Concurrent Computation of Quasi-Independent Layers (CQIL) can reduce latency by up to 48.3% on LLaMA-33B, while maintaining a close level of performance. |
| title | CQIL: Inference Latency Optimization with Concurrent Computation of Quasi-Independent Layers |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2404.06709 |