Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910691386982400 |
|---|---|
| author | Yi, Euiin Kim, Taehyeon Jeung, Hongseok Chang, Du-Seong Yun, Se-Young |
| author_facet | Yi, Euiin Kim, Taehyeon Jeung, Hongseok Chang, Du-Seong Yun, Se-Young |
| contents | Large language models (LLMs) have revolutionized natural language processing and broadened their applicability across diverse commercial applications. However, the deployment of these models is constrained by high inference time in multilingual settings. To mitigate this challenge, this paper explores a training recipe of an assistant model in speculative decoding, which is leveraged to draft and-then its future tokens are verified by the target LLM. We show that language-specific draft models, optimized through a targeted pretrain-and-finetune strategy, substantially brings a speedup in inference time compared to the previous methods. We validate these models across various languages in inference time, out-of-domain speedup, and GPT-4o evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_16758 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters Yi, Euiin Kim, Taehyeon Jeung, Hongseok Chang, Du-Seong Yun, Se-Young Computation and Language Large language models (LLMs) have revolutionized natural language processing and broadened their applicability across diverse commercial applications. However, the deployment of these models is constrained by high inference time in multilingual settings. To mitigate this challenge, this paper explores a training recipe of an assistant model in speculative decoding, which is leveraged to draft and-then its future tokens are verified by the target LLM. We show that language-specific draft models, optimized through a targeted pretrain-and-finetune strategy, substantially brings a speedup in inference time compared to the previous methods. We validate these models across various languages in inference time, out-of-domain speedup, and GPT-4o evaluation. |
| title | Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.16758 |