DynaMo: Accelerating Language Model Inference with Dynamic Multi-Token Sampling
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917655667015680 |
|---|---|
| author | Tuli, Shikhar Lin, Chi-Heng Hsu, Yen-Chang Jha, Niraj K. Shen, Yilin Jin, Hongxia |
| author_facet | Tuli, Shikhar Lin, Chi-Heng Hsu, Yen-Chang Jha, Niraj K. Shen, Yilin Jin, Hongxia |
| contents | Traditional language models operate autoregressively, i.e., they predict one token at a time. Rapid explosion in model sizes has resulted in high inference times. In this work, we propose DynaMo, a suite of multi-token prediction language models that reduce net inference times. Our models $\textit{dynamically}$ predict multiple tokens based on their confidence in the predicted joint probability distribution. We propose a lightweight technique to train these models, leveraging the weights of traditional autoregressive counterparts. Moreover, we propose novel ways to enhance the estimated joint probability to improve text generation quality, namely co-occurrence weighted masking and adaptive thresholding. We also propose systematic qualitative and quantitative methods to rigorously test the quality of generated text for non-autoregressive generation. One of the models in our suite, DynaMo-7.3B-T3, achieves same-quality generated text as the baseline (Pythia-6.9B) while achieving 2.57$\times$ speed-up with only 5.87% and 2.67% parameter and training time overheads, respectively. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_00888 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DynaMo: Accelerating Language Model Inference with Dynamic Multi-Token Sampling Tuli, Shikhar Lin, Chi-Heng Hsu, Yen-Chang Jha, Niraj K. Shen, Yilin Jin, Hongxia Computation and Language Traditional language models operate autoregressively, i.e., they predict one token at a time. Rapid explosion in model sizes has resulted in high inference times. In this work, we propose DynaMo, a suite of multi-token prediction language models that reduce net inference times. Our models $\textit{dynamically}$ predict multiple tokens based on their confidence in the predicted joint probability distribution. We propose a lightweight technique to train these models, leveraging the weights of traditional autoregressive counterparts. Moreover, we propose novel ways to enhance the estimated joint probability to improve text generation quality, namely co-occurrence weighted masking and adaptive thresholding. We also propose systematic qualitative and quantitative methods to rigorously test the quality of generated text for non-autoregressive generation. One of the models in our suite, DynaMo-7.3B-T3, achieves same-quality generated text as the baseline (Pythia-6.9B) while achieving 2.57$\times$ speed-up with only 5.87% and 2.67% parameter and training time overheads, respectively. |
| title | DynaMo: Accelerating Language Model Inference with Dynamic Multi-Token Sampling |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2405.00888 |