TRAWL: Tensor Reduced and Approximated Weights for Large Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915156269727744 |
|---|---|
| author | Luo, Yiran Patel, Het Fu, Yu Ahn, Dawon Chen, Jia Dong, Yue Papalexakis, Evangelos E. |
| author_facet | Luo, Yiran Patel, Het Fu, Yu Ahn, Dawon Chen, Jia Dong, Yue Papalexakis, Evangelos E. |
| contents | Recent research has shown that pruning large-scale language models for inference is an effective approach to improving model efficiency, significantly reducing model weights with minimal impact on performance. Interestingly, pruning can sometimes even enhance accuracy by removing noise that accumulates during training, particularly through matrix decompositions. However, recent work has primarily focused on single matrix decompositions or lower precision techniques, which may fail to fully capture structural patterns. To address these limitations, we introduce TRAWL (Tensor Reduced and Approximated Weights for Large Language Models), a technique that applies tensor decomposition across multiple weight matrices to effectively denoise LLMs by capturing global structural patterns. Our experiments show that TRAWL improves model performance by up to 16% over baseline models on benchmark datasets, without requiring additional data, training, or fine-tuning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_17261 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TRAWL: Tensor Reduced and Approximated Weights for Large Language Models Luo, Yiran Patel, Het Fu, Yu Ahn, Dawon Chen, Jia Dong, Yue Papalexakis, Evangelos E. Computation and Language 68T50 (Primary), 65F55 (Secondary) I.2.7 Recent research has shown that pruning large-scale language models for inference is an effective approach to improving model efficiency, significantly reducing model weights with minimal impact on performance. Interestingly, pruning can sometimes even enhance accuracy by removing noise that accumulates during training, particularly through matrix decompositions. However, recent work has primarily focused on single matrix decompositions or lower precision techniques, which may fail to fully capture structural patterns. To address these limitations, we introduce TRAWL (Tensor Reduced and Approximated Weights for Large Language Models), a technique that applies tensor decomposition across multiple weight matrices to effectively denoise LLMs by capturing global structural patterns. Our experiments show that TRAWL improves model performance by up to 16% over baseline models on benchmark datasets, without requiring additional data, training, or fine-tuning. |
| title | TRAWL: Tensor Reduced and Approximated Weights for Large Language Models |
| topic | Computation and Language 68T50 (Primary), 65F55 (Secondary) I.2.7 |
| url | https://arxiv.org/abs/2406.17261 |