CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909341510008832 |
|---|---|
| author | Bai, Yu Zou, Xiyuan Huang, Heyan Chen, Sanxing Rondeau, Marc-Antoine Gao, Yang Cheung, Jackie Chi Kit |
| author_facet | Bai, Yu Zou, Xiyuan Huang, Heyan Chen, Sanxing Rondeau, Marc-Antoine Gao, Yang Cheung, Jackie Chi Kit |
| contents | Long sequence modeling has gained broad interest as large language models (LLMs) continue to advance. Recent research has identified that a large portion of hidden states within the key-value caches of Transformer models can be discarded (also termed evicted) without affecting the perplexity performance in generating long sequences. However, we show that these methods, despite preserving perplexity performance, often drop information that is important for solving downstream tasks, a problem which we call information neglect. To address this issue, we introduce Chunked Instruction-aware State Eviction (CItruS), a novel modeling technique that integrates the attention preferences useful for a downstream task into the eviction process of hidden states. In addition, we design a method for chunked sequence processing to further improve efficiency. Our training-free method exhibits superior performance on long sequence comprehension and retrieval tasks over several strong baselines under the same memory budget, while preserving language modeling perplexity. The code and data have been released at https://github.com/ybai-nlp/CItruS. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_12018 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling Bai, Yu Zou, Xiyuan Huang, Heyan Chen, Sanxing Rondeau, Marc-Antoine Gao, Yang Cheung, Jackie Chi Kit Computation and Language Long sequence modeling has gained broad interest as large language models (LLMs) continue to advance. Recent research has identified that a large portion of hidden states within the key-value caches of Transformer models can be discarded (also termed evicted) without affecting the perplexity performance in generating long sequences. However, we show that these methods, despite preserving perplexity performance, often drop information that is important for solving downstream tasks, a problem which we call information neglect. To address this issue, we introduce Chunked Instruction-aware State Eviction (CItruS), a novel modeling technique that integrates the attention preferences useful for a downstream task into the eviction process of hidden states. In addition, we design a method for chunked sequence processing to further improve efficiency. Our training-free method exhibits superior performance on long sequence comprehension and retrieval tasks over several strong baselines under the same memory budget, while preserving language modeling perplexity. The code and data have been released at https://github.com/ybai-nlp/CItruS. |
| title | CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.12018 |