Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.22581 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910033692852224 |
|---|---|
| author | Bian, Tian Niu, Yifan Yuan, Chaohao Piao, Chengzhi Wu, Bingzhe Huang, Long-Kai Rong, Yu Xu, Tingyang Cheng, Hong Li, Jia |
| author_facet | Bian, Tian Niu, Yifan Yuan, Chaohao Piao, Chengzhi Wu, Bingzhe Huang, Long-Kai Rong, Yu Xu, Tingyang Cheng, Hong Li, Jia |
| contents | Circuit discovery has recently attracted attention as a potential research direction to explain the non-trivial behaviors of language models. It aims to find the computational subgraphs, also known as circuits, within the model that are responsible for solving specific tasks. However, most existing studies overlook the holistic nature of these circuits and require designing specific corrupted activations for different tasks, which is inaccurate and inefficient. In this work, we propose an end-to-end approach based on the principle of Information Bottleneck, called IBCircuit, to identify informative circuits holistically. IBCircuit is an optimization framework for holistic circuit discovery and can be applied to any given task without tediously corrupted activation design. In both the Indirect Object Identification (IOI) and Greater-Than tasks, IBCircuit identifies more faithful and minimal circuits in terms of critical node components and edge components compared to recent related work. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_22581 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck Bian, Tian Niu, Yifan Yuan, Chaohao Piao, Chengzhi Wu, Bingzhe Huang, Long-Kai Rong, Yu Xu, Tingyang Cheng, Hong Li, Jia Machine Learning Circuit discovery has recently attracted attention as a potential research direction to explain the non-trivial behaviors of language models. It aims to find the computational subgraphs, also known as circuits, within the model that are responsible for solving specific tasks. However, most existing studies overlook the holistic nature of these circuits and require designing specific corrupted activations for different tasks, which is inaccurate and inefficient. In this work, we propose an end-to-end approach based on the principle of Information Bottleneck, called IBCircuit, to identify informative circuits holistically. IBCircuit is an optimization framework for holistic circuit discovery and can be applied to any given task without tediously corrupted activation design. In both the Indirect Object Identification (IOI) and Greater-Than tasks, IBCircuit identifies more faithful and minimal circuits in terms of critical node components and edge components compared to recent related work. |
| title | IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2602.22581 |