UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compression
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912594590171136 |
|---|---|
| author | Deng, Chenlong Zhang, Zhisong Mao, Kelong Li, Shuaiyi Fang, Tianqing Zhang, Hongming Mi, Haitao Yu, Dong Dou, Zhicheng |
| author_facet | Deng, Chenlong Zhang, Zhisong Mao, Kelong Li, Shuaiyi Fang, Tianqing Zhang, Hongming Mi, Haitao Yu, Dong Dou, Zhicheng |
| contents | Large language models are increasingly capable of handling long-context inputs, but the memory overhead of key-value (KV) cache remains a major bottleneck for general-purpose deployment. While various compression strategies have been explored, sequence-level compression, which drops the full KV caches for certain tokens, is particularly challenging as it can lead to the loss of important contextual information. To address this, we introduce UniGist, a sequence-level long-context compression framework that efficiently preserves context information by replacing raw tokens with special compression tokens (gists) in a fine-grained manner. We adopt a chunk-free training strategy and design an efficient kernel with a gist shift trick, enabling optimized GPU training. Our scheme also supports flexible inference by allowing the actual removal of compressed tokens, resulting in real-time memory savings. Experiments across multiple long-context tasks demonstrate that UniGist significantly improves compression quality, with especially strong performance in detail-recalling tasks and long-range dependency modeling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_15763 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compression Deng, Chenlong Zhang, Zhisong Mao, Kelong Li, Shuaiyi Fang, Tianqing Zhang, Hongming Mi, Haitao Yu, Dong Dou, Zhicheng Computation and Language Large language models are increasingly capable of handling long-context inputs, but the memory overhead of key-value (KV) cache remains a major bottleneck for general-purpose deployment. While various compression strategies have been explored, sequence-level compression, which drops the full KV caches for certain tokens, is particularly challenging as it can lead to the loss of important contextual information. To address this, we introduce UniGist, a sequence-level long-context compression framework that efficiently preserves context information by replacing raw tokens with special compression tokens (gists) in a fine-grained manner. We adopt a chunk-free training strategy and design an efficient kernel with a gist shift trick, enabling optimized GPU training. Our scheme also supports flexible inference by allowing the actual removal of compressed tokens, resulting in real-time memory savings. Experiments across multiple long-context tasks demonstrate that UniGist significantly improves compression quality, with especially strong performance in detail-recalling tasks and long-range dependency modeling. |
| title | UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compression |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2509.15763 |