Attn-GS: Attention-Guided Context Compression for Efficient Personalized LLMs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918327235903488 |
|---|---|
| author | Zeng, Shenglai Zheng, Tianqi Tian, Chuan Everaert, Dante Wang, Yau-Shian Huang, Yupin Morais, Michael J. Patki, Rohit Tian, Jinjin Dai, Xinnan Guo, Kai Cheng, Monica Xiao Liu, Hui |
| author_facet | Zeng, Shenglai Zheng, Tianqi Tian, Chuan Everaert, Dante Wang, Yau-Shian Huang, Yupin Morais, Michael J. Patki, Rohit Tian, Jinjin Dai, Xinnan Guo, Kai Cheng, Monica Xiao Liu, Hui |
| contents | Personalizing large language models (LLMs) to individual users requires incorporating extensive interaction histories and profiles, but input token constraints make this impractical due to high inference latency and API costs. Existing approaches rely on heuristic methods such as selecting recent interactions or prompting summarization models to compress user profiles. However, these methods treat context as a monolithic whole and fail to consider how LLMs internally process and prioritize different profile components. We investigate whether LLMs' attention patterns can effectively identify important personalization signals for intelligent context compression. Through preliminary studies on representative personalization tasks, we discover that (a) LLMs' attention patterns naturally reveal important signals, and (b) fine-tuning enhances LLMs' ability to distinguish between relevant and irrelevant information. Based on these insights, we propose Attn-GS, an attention-guided context compression framework that leverages attention feedback from a marking model to mark important personalization sentences, then guides a compression model to generate task-relevant, high-quality compressed user contexts. Extensive experiments demonstrate that Attn-GS significantly outperforms various baselines across different tasks, token limits, and settings, achieving performance close to using full context while reducing token usage by 50 times. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_07778 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Attn-GS: Attention-Guided Context Compression for Efficient Personalized LLMs Zeng, Shenglai Zheng, Tianqi Tian, Chuan Everaert, Dante Wang, Yau-Shian Huang, Yupin Morais, Michael J. Patki, Rohit Tian, Jinjin Dai, Xinnan Guo, Kai Cheng, Monica Xiao Liu, Hui Computation and Language Personalizing large language models (LLMs) to individual users requires incorporating extensive interaction histories and profiles, but input token constraints make this impractical due to high inference latency and API costs. Existing approaches rely on heuristic methods such as selecting recent interactions or prompting summarization models to compress user profiles. However, these methods treat context as a monolithic whole and fail to consider how LLMs internally process and prioritize different profile components. We investigate whether LLMs' attention patterns can effectively identify important personalization signals for intelligent context compression. Through preliminary studies on representative personalization tasks, we discover that (a) LLMs' attention patterns naturally reveal important signals, and (b) fine-tuning enhances LLMs' ability to distinguish between relevant and irrelevant information. Based on these insights, we propose Attn-GS, an attention-guided context compression framework that leverages attention feedback from a marking model to mark important personalization sentences, then guides a compression model to generate task-relevant, high-quality compressed user contexts. Extensive experiments demonstrate that Attn-GS significantly outperforms various baselines across different tasks, token limits, and settings, achieving performance close to using full context while reducing token usage by 50 times. |
| title | Attn-GS: Attention-Guided Context Compression for Efficient Personalized LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2602.07778 |