From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908997311791104 |
|---|---|
| author | Shi, Ling Wu, Xinwei Zhao, Xiaohu Wang, Hao Liu, Heng Liu, Yangyang Xu, Linlong Wang, Longyue Xiong, Deyi Luo, Weihua |
| author_facet | Shi, Ling Wu, Xinwei Zhao, Xiaohu Wang, Hao Liu, Heng Liu, Yangyang Xu, Linlong Wang, Longyue Xiong, Deyi Luo, Weihua |
| contents | While mechanistic interpretability tools like Sparse Autoencoders (SAEs) can uncover meaningful features within Large Language Models (LLMs), a critical gap remains in transforming these insights into practical actions for model optimization. We bridge this gap with the hypothesis that data selection guided by a model's internal task features is a effective training strategy. Inspired by this, we propose Interpretability-Guided Data Selection (IGDS), a framework that first identifies these causal task features through frequency recall and interventional filtering, then selects ``Feature-Resonant Data'' that maximally activates task features for fine-tuning. We validate IGDS on mathematical reasoning, summarization, and translation tasks within Gemma-2, LLaMA-3.1, and Qwen3 models. Our experiments demonstrate exceptional data efficiency: on the Math task, IGDS surpasses full-dataset fine-tuning by a remarkable 17.4% on Gemma-2-2B while using only 50% of the data, and outperforms established baselines focused on data quality and diversity. Analysis confirms a strong positive correlation between feature amplification and task performance improvement. IGDS thus provides a direct and effective framework to enhance LLMs by leveraging their internal mechanisms, validating our core hypothesis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_25167 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models Shi, Ling Wu, Xinwei Zhao, Xiaohu Wang, Hao Liu, Heng Liu, Yangyang Xu, Linlong Wang, Longyue Xiong, Deyi Luo, Weihua Artificial Intelligence While mechanistic interpretability tools like Sparse Autoencoders (SAEs) can uncover meaningful features within Large Language Models (LLMs), a critical gap remains in transforming these insights into practical actions for model optimization. We bridge this gap with the hypothesis that data selection guided by a model's internal task features is a effective training strategy. Inspired by this, we propose Interpretability-Guided Data Selection (IGDS), a framework that first identifies these causal task features through frequency recall and interventional filtering, then selects ``Feature-Resonant Data'' that maximally activates task features for fine-tuning. We validate IGDS on mathematical reasoning, summarization, and translation tasks within Gemma-2, LLaMA-3.1, and Qwen3 models. Our experiments demonstrate exceptional data efficiency: on the Math task, IGDS surpasses full-dataset fine-tuning by a remarkable 17.4% on Gemma-2-2B while using only 50% of the data, and outperforms established baselines focused on data quality and diversity. Analysis confirms a strong positive correlation between feature amplification and task performance improvement. IGDS thus provides a direct and effective framework to enhance LLMs by leveraging their internal mechanisms, validating our core hypothesis. |
| title | From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2604.25167 |