Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2404.11502 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911844188291072 |
|---|---|
| author | Chen, Yushuo Tang, Tianyi Xiang, Erge Li, Linjiang Zhao, Wayne Xin Wang, Jing Chai, Yunpeng Wen, Ji-Rong |
| author_facet | Chen, Yushuo Tang, Tianyi Xiang, Erge Li, Linjiang Zhao, Wayne Xin Wang, Jing Chai, Yunpeng Wen, Ji-Rong |
| contents | In real world, large language models (LLMs) can serve as the assistant to help users accomplish their jobs, and also support the development of advanced applications. For the wide application of LLMs, the inference efficiency is an essential concern, which has been widely studied in existing work, and numerous optimization algorithms and code libraries have been proposed to improve it. Nonetheless, users still find it challenging to compare the effectiveness of all the above methods and understand the underlying mechanisms. In this work, we perform a detailed coarse-to-fine analysis of the inference performance of various code libraries. To evaluate the overall effectiveness, we examine four usage scenarios within two practical applications. We further provide both theoretical and empirical fine-grained analyses of each module in the Transformer architecture. Our experiments yield comprehensive results that are invaluable for researchers to evaluate code libraries and improve inference strategies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_11502 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Coarse-to-Fine Evaluation of Inference Efficiency for Large Language Models Chen, Yushuo Tang, Tianyi Xiang, Erge Li, Linjiang Zhao, Wayne Xin Wang, Jing Chai, Yunpeng Wen, Ji-Rong Computation and Language Artificial Intelligence In real world, large language models (LLMs) can serve as the assistant to help users accomplish their jobs, and also support the development of advanced applications. For the wide application of LLMs, the inference efficiency is an essential concern, which has been widely studied in existing work, and numerous optimization algorithms and code libraries have been proposed to improve it. Nonetheless, users still find it challenging to compare the effectiveness of all the above methods and understand the underlying mechanisms. In this work, we perform a detailed coarse-to-fine analysis of the inference performance of various code libraries. To evaluate the overall effectiveness, we examine four usage scenarios within two practical applications. We further provide both theoretical and empirical fine-grained analyses of each module in the Transformer architecture. Our experiments yield comprehensive results that are invaluable for researchers to evaluate code libraries and improve inference strategies. |
| title | Towards Coarse-to-Fine Evaluation of Inference Efficiency for Large Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2404.11502 |