Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.03779 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912613773869056 |
|---|---|
| author | Yu, Lei Niu, Jingcheng Zhu, Zining Chen, Xi Penn, Gerald |
| author_facet | Yu, Lei Niu, Jingcheng Zhu, Zining Chen, Xi Penn, Gerald |
| contents | In this paper, we introduce DiscoGP, a novel framework for extracting self-contained modular units, or sheaves, within neural language models (LMs). Sheaves extend the concept of functional circuits, a unit widely explored in interpretability research, by considering not only subsets of edges in an LM's computation graph but also the model's weight parameters. Our framework identifies sheaves through a gradient-based pruning algorithm that operates on both of these in such a way that reduces the original LM to a sparse skeleton that preserves certain core capabilities. Experimental results demonstrate that, across a range of linguistic and reasoning tasks, DiscoGP extracts sheaves that preserve 93%-100% of a model's performance on the identified task while comprising only 1%-7% of the original weights and connections. Furthermore, our analysis reveals that, compared to previously identified LM circuits, the sheaves discovered by DiscoGP exhibit superior modularity and functional fidelity. Extending our method to the neuron level also unveils novel insights into the inner workings of LLMs |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_03779 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Sheaf Discovery with Joint Computation Graph Pruning and Flexible Granularity Yu, Lei Niu, Jingcheng Zhu, Zining Chen, Xi Penn, Gerald Computation and Language Machine Learning In this paper, we introduce DiscoGP, a novel framework for extracting self-contained modular units, or sheaves, within neural language models (LMs). Sheaves extend the concept of functional circuits, a unit widely explored in interpretability research, by considering not only subsets of edges in an LM's computation graph but also the model's weight parameters. Our framework identifies sheaves through a gradient-based pruning algorithm that operates on both of these in such a way that reduces the original LM to a sparse skeleton that preserves certain core capabilities. Experimental results demonstrate that, across a range of linguistic and reasoning tasks, DiscoGP extracts sheaves that preserve 93%-100% of a model's performance on the identified task while comprising only 1%-7% of the original weights and connections. Furthermore, our analysis reveals that, compared to previously identified LM circuits, the sheaves discovered by DiscoGP exhibit superior modularity and functional fidelity. Extending our method to the neuron level also unveils novel insights into the inner workings of LLMs |
| title | Sheaf Discovery with Joint Computation Graph Pruning and Flexible Granularity |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2407.03779 |