FutureFill: Fast Generation from Convolutional Sequence Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913905771544576 |
|---|---|
| author | Agarwal, Naman Chen, Xinyi Dogariu, Evan Shah, Devan Strauss, Hubert Feinberg, Vlad Suo, Daniel Bartlett, Peter Hazan, Elad |
| author_facet | Agarwal, Naman Chen, Xinyi Dogariu, Evan Shah, Devan Strauss, Hubert Feinberg, Vlad Suo, Daniel Bartlett, Peter Hazan, Elad |
| contents | We address the challenge of efficient auto-regressive generation in sequence prediction models by introducing FutureFill, a general-purpose fast generation method for any sequence prediction algorithm based on convolutional operators. FutureFill reduces generation time from quadratic to quasilinear in the context length. Moreover, when generating from a prompt, it requires a prefill cache whose size grows only with the number of tokens to be generated, often much smaller than the caches required by standard convolutional or attention based models. We validate our theoretical claims with experiments on synthetic tasks and demonstrate substantial efficiency gains when generating from a deep convolutional sequence prediction model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_03766 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FutureFill: Fast Generation from Convolutional Sequence Models Agarwal, Naman Chen, Xinyi Dogariu, Evan Shah, Devan Strauss, Hubert Feinberg, Vlad Suo, Daniel Bartlett, Peter Hazan, Elad Machine Learning Artificial Intelligence Computation and Language We address the challenge of efficient auto-regressive generation in sequence prediction models by introducing FutureFill, a general-purpose fast generation method for any sequence prediction algorithm based on convolutional operators. FutureFill reduces generation time from quadratic to quasilinear in the context length. Moreover, when generating from a prompt, it requires a prefill cache whose size grows only with the number of tokens to be generated, often much smaller than the caches required by standard convolutional or attention based models. We validate our theoretical claims with experiments on synthetic tasks and demonstrate substantial efficiency gains when generating from a deep convolutional sequence prediction model. |
| title | FutureFill: Fast Generation from Convolutional Sequence Models |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2410.03766 |