Change Is the Only Constant: Dynamic LLM Slicing based on Layer Redundancy
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912106894327808 |
|---|---|
| author | Dumitru, Razvan-Gabriel Clotan, Paul-Ioan Yadav, Vikas Peteleaza, Darius Surdeanu, Mihai |
| author_facet | Dumitru, Razvan-Gabriel Clotan, Paul-Ioan Yadav, Vikas Peteleaza, Darius Surdeanu, Mihai |
| contents | This paper introduces a novel model compression approach through dynamic layer-specific pruning in Large Language Models (LLMs), enhancing the traditional methodology established by SliceGPT. By transitioning from constant to dynamic slicing, our method leverages the newly proposed Layer Redundancy (LR) score, which assesses how much change each layer changes its input by measuring the cosine similarity of the input to the output of the layer. We use this score to prune parts of individual layers based on redundancy in such a way that the average pruned percentage for all layers is a fixed value. We conducted extensive experiments using models like Llama3-8B and Mistral-7B on multiple datasets, evaluating different slicing bases and percentages to determine optimal configurations that balance efficiency and performance. Our findings show that our dynamic slicing approach not only maintains but, in many cases, enhances model performance compared to the baseline established by constant slicing methods. For instance, in several settings, we see performance improvements of up to 5% over the SliceGPT baseline. Additionally, a perplexity decrease by as much as 7% was observed across multiple benchmarks, validating the effectiveness of our method. The code, model weights, and datasets are open-sourced at https://github.com/RazvanDu/DynamicSlicing. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_03513 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Change Is the Only Constant: Dynamic LLM Slicing based on Layer Redundancy Dumitru, Razvan-Gabriel Clotan, Paul-Ioan Yadav, Vikas Peteleaza, Darius Surdeanu, Mihai Computation and Language Artificial Intelligence Machine Learning I.2.7; I.2.0 This paper introduces a novel model compression approach through dynamic layer-specific pruning in Large Language Models (LLMs), enhancing the traditional methodology established by SliceGPT. By transitioning from constant to dynamic slicing, our method leverages the newly proposed Layer Redundancy (LR) score, which assesses how much change each layer changes its input by measuring the cosine similarity of the input to the output of the layer. We use this score to prune parts of individual layers based on redundancy in such a way that the average pruned percentage for all layers is a fixed value. We conducted extensive experiments using models like Llama3-8B and Mistral-7B on multiple datasets, evaluating different slicing bases and percentages to determine optimal configurations that balance efficiency and performance. Our findings show that our dynamic slicing approach not only maintains but, in many cases, enhances model performance compared to the baseline established by constant slicing methods. For instance, in several settings, we see performance improvements of up to 5% over the SliceGPT baseline. Additionally, a perplexity decrease by as much as 7% was observed across multiple benchmarks, validating the effectiveness of our method. The code, model weights, and datasets are open-sourced at https://github.com/RazvanDu/DynamicSlicing. |
| title | Change Is the Only Constant: Dynamic LLM Slicing based on Layer Redundancy |
| topic | Computation and Language Artificial Intelligence Machine Learning I.2.7; I.2.0 |
| url | https://arxiv.org/abs/2411.03513 |