QUITO-X: A New Perspective on Context Compression from the Information Bottleneck Theory
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918158858715136 |
|---|---|
| author | Wang, Yihang Huang, Xu Tian, Bowen Su, Yueyang Yu, Lei Liao, Huaming Fan, Yixing Guo, Jiafeng Cheng, Xueqi |
| author_facet | Wang, Yihang Huang, Xu Tian, Bowen Su, Yueyang Yu, Lei Liao, Huaming Fan, Yixing Guo, Jiafeng Cheng, Xueqi |
| contents | Generative LLM have achieved remarkable success in various industrial applications, owing to their promising In-Context Learning capabilities. However, the issue of long context in complex tasks poses a significant barrier to their wider adoption, manifested in two main aspects: (i) The excessively long context leads to high costs and inference delays. (ii) A substantial amount of task-irrelevant information introduced by long contexts exacerbates the "lost in the middle" problem. Existing methods compress context by removing redundant tokens using metrics such as self-information or PPL, which is inconsistent with the objective of retaining the most important tokens when conditioning on a given query. In this study, we introduce information bottleneck theory (IB) to model the problem, offering a novel perspective that thoroughly addresses the essential properties required for context compression. Additionally, we propose a cross-attention-based approach to approximate mutual information in IB, which can be flexibly replaced with suitable alternatives in different scenarios. Extensive experiments on four datasets demonstrate that our method achieves a 25% increase in compression rate compared to the state-of-the-art, while maintaining question answering performance. In particular, the context compressed by our method even outperform the full context in some cases. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_10497 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | QUITO-X: A New Perspective on Context Compression from the Information Bottleneck Theory Wang, Yihang Huang, Xu Tian, Bowen Su, Yueyang Yu, Lei Liao, Huaming Fan, Yixing Guo, Jiafeng Cheng, Xueqi Computation and Language Artificial Intelligence Generative LLM have achieved remarkable success in various industrial applications, owing to their promising In-Context Learning capabilities. However, the issue of long context in complex tasks poses a significant barrier to their wider adoption, manifested in two main aspects: (i) The excessively long context leads to high costs and inference delays. (ii) A substantial amount of task-irrelevant information introduced by long contexts exacerbates the "lost in the middle" problem. Existing methods compress context by removing redundant tokens using metrics such as self-information or PPL, which is inconsistent with the objective of retaining the most important tokens when conditioning on a given query. In this study, we introduce information bottleneck theory (IB) to model the problem, offering a novel perspective that thoroughly addresses the essential properties required for context compression. Additionally, we propose a cross-attention-based approach to approximate mutual information in IB, which can be flexibly replaced with suitable alternatives in different scenarios. Extensive experiments on four datasets demonstrate that our method achieves a 25% increase in compression rate compared to the state-of-the-art, while maintaining question answering performance. In particular, the context compressed by our method even outperform the full context in some cases. |
| title | QUITO-X: A New Perspective on Context Compression from the Information Bottleneck Theory |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2408.10497 |