SPIN: Sparsifying and Integrating Internal Neurons in Large Language Models for Text Classification
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916275393921024 |
|---|---|
| author | Jiao, Difan Liu, Yilun Tang, Zhenwei Matter, Daniel Pfeffer, Jürgen Anderson, Ashton |
| author_facet | Jiao, Difan Liu, Yilun Tang, Zhenwei Matter, Daniel Pfeffer, Jürgen Anderson, Ashton |
| contents | Among the many tasks that Large Language Models (LLMs) have revolutionized is text classification. Current text classification paradigms, however, rely solely on the output of the final layer in the LLM, with the rich information contained in internal neurons largely untapped. In this study, we present SPIN: a model-agnostic framework that sparsifies and integrates internal neurons of intermediate layers of LLMs for text classification. Specifically, SPIN sparsifies internal neurons by linear probing-based salient neuron selection layer by layer, avoiding noise from unrelated neurons and ensuring efficiency. The cross-layer salient neurons are then integrated to serve as multi-layered features for the classification head. Extensive experimental results show our proposed SPIN significantly improves text classification accuracy, efficiency, and interpretability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_15983 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | SPIN: Sparsifying and Integrating Internal Neurons in Large Language Models for Text Classification Jiao, Difan Liu, Yilun Tang, Zhenwei Matter, Daniel Pfeffer, Jürgen Anderson, Ashton Machine Learning Artificial Intelligence Computation and Language Among the many tasks that Large Language Models (LLMs) have revolutionized is text classification. Current text classification paradigms, however, rely solely on the output of the final layer in the LLM, with the rich information contained in internal neurons largely untapped. In this study, we present SPIN: a model-agnostic framework that sparsifies and integrates internal neurons of intermediate layers of LLMs for text classification. Specifically, SPIN sparsifies internal neurons by linear probing-based salient neuron selection layer by layer, avoiding noise from unrelated neurons and ensuring efficiency. The cross-layer salient neurons are then integrated to serve as multi-layered features for the classification head. Extensive experimental results show our proposed SPIN significantly improves text classification accuracy, efficiency, and interpretability. |
| title | SPIN: Sparsifying and Integrating Internal Neurons in Large Language Models for Text Classification |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2311.15983 |