First Activations Matter: Training-Free Methods for Dynamic Activation in Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909292927385600 |
|---|---|
| author | Ma, Chi Huang, Mincong Zhang, Ying Wang, Chao Wang, Yujie Yu, Lei Liu, Chuan Lin, Wei |
| author_facet | Ma, Chi Huang, Mincong Zhang, Ying Wang, Chao Wang, Yujie Yu, Lei Liu, Chuan Lin, Wei |
| contents | Dynamic activation (DA) techniques, such as DejaVu and MoEfication, have demonstrated their potential to significantly enhance the inference efficiency of large language models (LLMs). However, these techniques often rely on ReLU activation functions or require additional parameters and training to maintain performance. This paper introduces a training-free Threshold-based Dynamic Activation(TDA) method that leverage sequence information to exploit the inherent sparsity of models across various architectures. This method is designed to accelerate generation speed by 18-25\% without significantly compromising task performance, thereby addressing the limitations of existing DA techniques. Moreover, we delve into the root causes of LLM sparsity and theoretically analyze two of its critical features: history-related activation uncertainty and semantic-irrelevant activation inertia. Our comprehensive analyses not only provide a robust theoretical foundation for DA methods but also offer valuable insights to guide future research in optimizing LLMs for greater efficiency and effectiveness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_11393 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | First Activations Matter: Training-Free Methods for Dynamic Activation in Large Language Models Ma, Chi Huang, Mincong Zhang, Ying Wang, Chao Wang, Yujie Yu, Lei Liu, Chuan Lin, Wei Computation and Language Machine Learning Dynamic activation (DA) techniques, such as DejaVu and MoEfication, have demonstrated their potential to significantly enhance the inference efficiency of large language models (LLMs). However, these techniques often rely on ReLU activation functions or require additional parameters and training to maintain performance. This paper introduces a training-free Threshold-based Dynamic Activation(TDA) method that leverage sequence information to exploit the inherent sparsity of models across various architectures. This method is designed to accelerate generation speed by 18-25\% without significantly compromising task performance, thereby addressing the limitations of existing DA techniques. Moreover, we delve into the root causes of LLM sparsity and theoretically analyze two of its critical features: history-related activation uncertainty and semantic-irrelevant activation inertia. Our comprehensive analyses not only provide a robust theoretical foundation for DA methods but also offer valuable insights to guide future research in optimizing LLMs for greater efficiency and effectiveness. |
| title | First Activations Matter: Training-Free Methods for Dynamic Activation in Large Language Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2408.11393 |