Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2402.16844 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911957670428672 |
|---|---|
| author | Bergner, Benjamin Skliar, Andrii Royer, Amelie Blankevoort, Tijmen Asano, Yuki Bejnordi, Babak Ehteshami |
| author_facet | Bergner, Benjamin Skliar, Andrii Royer, Amelie Blankevoort, Tijmen Asano, Yuki Bejnordi, Babak Ehteshami |
| contents | Large language models (LLMs) have become ubiquitous in practice and are widely used for generation tasks such as translation, summarization and instruction following. However, their enormous size and reliance on autoregressive decoding increase deployment costs and complicate their use in latency-critical applications. In this work, we propose a hybrid approach that combines language models of different sizes to increase the efficiency of autoregressive decoding while maintaining high performance. Our method utilizes a pretrained frozen LLM that encodes all prompt tokens once in parallel, and uses the resulting representations to condition and guide a small language model (SLM), which then generates the response more efficiently. We investigate the combination of encoder-decoder LLMs with both encoder-decoder and decoder-only SLMs from different model families and only require fine-tuning of the SLM. Experiments with various benchmarks show substantial speedups of up to $4\times$, with minor performance penalties of $1-2\%$ for translation and summarization tasks compared to the LLM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_16844 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Think Big, Generate Quick: LLM-to-SLM for Fast Autoregressive Decoding Bergner, Benjamin Skliar, Andrii Royer, Amelie Blankevoort, Tijmen Asano, Yuki Bejnordi, Babak Ehteshami Machine Learning Artificial Intelligence Computation and Language Large language models (LLMs) have become ubiquitous in practice and are widely used for generation tasks such as translation, summarization and instruction following. However, their enormous size and reliance on autoregressive decoding increase deployment costs and complicate their use in latency-critical applications. In this work, we propose a hybrid approach that combines language models of different sizes to increase the efficiency of autoregressive decoding while maintaining high performance. Our method utilizes a pretrained frozen LLM that encodes all prompt tokens once in parallel, and uses the resulting representations to condition and guide a small language model (SLM), which then generates the response more efficiently. We investigate the combination of encoder-decoder LLMs with both encoder-decoder and decoder-only SLMs from different model families and only require fine-tuning of the SLM. Experiments with various benchmarks show substantial speedups of up to $4\times$, with minor performance penalties of $1-2\%$ for translation and summarization tasks compared to the LLM. |
| title | Think Big, Generate Quick: LLM-to-SLM for Fast Autoregressive Decoding |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2402.16844 |