Eigenpruning: an Interpretability-Inspired PEFT Method
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910494955143168 |
|---|---|
| author | Vergara-Browne, Tomás Soto, Álvaro Aizawa, Akiko |
| author_facet | Vergara-Browne, Tomás Soto, Álvaro Aizawa, Akiko |
| contents | We introduce eigenpruning, a method that removes singular values from weight matrices in an LLM to improve its performance in a particular task. This method is inspired by interpretability methods designed to automatically find subnetworks of a model which solve a specific task. In our tests, the pruned model outperforms the original model by a large margin, while only requiring minimal computation to prune the weight matrices. In the case of a small synthetic task in integer multiplication, the Phi-2 model can improve its accuracy in the test set from 13.75% to 97.50%. Interestingly, these results seem to indicate the existence of a computation path that can solve the task very effectively, but it was not being used by the original model. Finally, we publicly release our implementation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_03147 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Eigenpruning: an Interpretability-Inspired PEFT Method Vergara-Browne, Tomás Soto, Álvaro Aizawa, Akiko Machine Learning Artificial Intelligence We introduce eigenpruning, a method that removes singular values from weight matrices in an LLM to improve its performance in a particular task. This method is inspired by interpretability methods designed to automatically find subnetworks of a model which solve a specific task. In our tests, the pruned model outperforms the original model by a large margin, while only requiring minimal computation to prune the weight matrices. In the case of a small synthetic task in integer multiplication, the Phi-2 model can improve its accuracy in the test set from 13.75% to 97.50%. Interestingly, these results seem to indicate the existence of a computation path that can solve the task very effectively, but it was not being used by the original model. Finally, we publicly release our implementation. |
| title | Eigenpruning: an Interpretability-Inspired PEFT Method |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2404.03147 |