Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2305.15805 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913371536752640 |
|---|---|
| author | Anagnostidis, Sotiris Pavllo, Dario Biggio, Luca Noci, Lorenzo Lucchi, Aurelien Hofmann, Thomas |
| author_facet | Anagnostidis, Sotiris Pavllo, Dario Biggio, Luca Noci, Lorenzo Lucchi, Aurelien Hofmann, Thomas |
| contents | Autoregressive Transformers adopted in Large Language Models (LLMs) are hard to scale to long sequences. Despite several works trying to reduce their computational cost, most of LLMs still adopt attention layers between all pairs of tokens in the sequence, thus incurring a quadratic cost. In this study, we present a novel approach that dynamically prunes contextual information while preserving the model's expressiveness, resulting in reduced memory and computational requirements during inference. Our method employs a learnable mechanism that determines which uninformative tokens can be dropped from the context at any point across the generation process. By doing so, our approach not only addresses performance concerns but also enhances interpretability, providing valuable insight into the model's decision-making process. Our technique can be applied to existing pre-trained models through a straightforward fine-tuning process, and the pruning strength can be specified by a sparsity parameter. Notably, our empirical findings demonstrate that we can effectively prune up to 80\% of the context without significant performance degradation on downstream tasks, offering a valuable tool for mitigating inference costs. Our reference implementation achieves up to $2\times$ increase in inference throughput and even greater memory savings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_15805 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers Anagnostidis, Sotiris Pavllo, Dario Biggio, Luca Noci, Lorenzo Lucchi, Aurelien Hofmann, Thomas Computation and Language Machine Learning Autoregressive Transformers adopted in Large Language Models (LLMs) are hard to scale to long sequences. Despite several works trying to reduce their computational cost, most of LLMs still adopt attention layers between all pairs of tokens in the sequence, thus incurring a quadratic cost. In this study, we present a novel approach that dynamically prunes contextual information while preserving the model's expressiveness, resulting in reduced memory and computational requirements during inference. Our method employs a learnable mechanism that determines which uninformative tokens can be dropped from the context at any point across the generation process. By doing so, our approach not only addresses performance concerns but also enhances interpretability, providing valuable insight into the model's decision-making process. Our technique can be applied to existing pre-trained models through a straightforward fine-tuning process, and the pruning strength can be specified by a sparsity parameter. Notably, our empirical findings demonstrate that we can effectively prune up to 80\% of the context without significant performance degradation on downstream tasks, offering a valuable tool for mitigating inference costs. Our reference implementation achieves up to $2\times$ increase in inference throughput and even greater memory savings. |
| title | Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2305.15805 |