Sentinel: Decoding Context Utilization via Attention Probing for Efficient LLM Context Compression
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908784164601856 |
|---|---|
| author | Zhang, Yong Li, Heng Huang, Yanwen Cheng, Ning Guo, Yang Zhu, Yun Wang, Yanmeng Wang, Shaojun Xiao, Jing |
| author_facet | Zhang, Yong Li, Heng Huang, Yanwen Cheng, Ning Guo, Yang Zhu, Yun Wang, Yanmeng Wang, Shaojun Xiao, Jing |
| contents | Retrieval-augmented generation (RAG) often suffers from long and noisy retrieved contexts. Prior context compression methods rely on predefined importance metrics or supervised compression models, rather than on the model's own inference-time behavior. We propose Sentinel, a lightweight sentence-level compression framework that treats context compression as an understanding decoding problem. Sentinel probes native attention behaviors of a frozen LLM with a lightweight readout to decode which parts of the context are actually utilized when answering a query, rather than using attention as a direct relevance score. We empirically observe that decoded relevance signals exhibit sufficient consistency across model scales to support effective compression with compact proxy models. On LongBench, Sentinel with a 0.5B proxy model achieves up to 5x compression while matching the QA performance of 7B-scale baselines, and despite being trained only on English QA data, generalizes effectively to Chinese and out-of-domain settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_23277 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Sentinel: Decoding Context Utilization via Attention Probing for Efficient LLM Context Compression Zhang, Yong Li, Heng Huang, Yanwen Cheng, Ning Guo, Yang Zhu, Yun Wang, Yanmeng Wang, Shaojun Xiao, Jing Computation and Language Artificial Intelligence Retrieval-augmented generation (RAG) often suffers from long and noisy retrieved contexts. Prior context compression methods rely on predefined importance metrics or supervised compression models, rather than on the model's own inference-time behavior. We propose Sentinel, a lightweight sentence-level compression framework that treats context compression as an understanding decoding problem. Sentinel probes native attention behaviors of a frozen LLM with a lightweight readout to decode which parts of the context are actually utilized when answering a query, rather than using attention as a direct relevance score. We empirically observe that decoded relevance signals exhibit sufficient consistency across model scales to support effective compression with compact proxy models. On LongBench, Sentinel with a 0.5B proxy model achieves up to 5x compression while matching the QA performance of 7B-scale baselines, and despite being trained only on English QA data, generalizes effectively to Chinese and out-of-domain settings. |
| title | Sentinel: Decoding Context Utilization via Attention Probing for Efficient LLM Context Compression |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2505.23277 |