DAST: Context-Aware Compression in LLMs via Dynamic Allocation of Soft Tokens
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909496701353984 |
|---|---|
| author | Chen, Shaoshen Li, Yangning Xu, Zishan Li, Yinghui Su, Xin Shan, Zifei Zheng, Hai-tao |
| author_facet | Chen, Shaoshen Li, Yangning Xu, Zishan Li, Yinghui Su, Xin Shan, Zifei Zheng, Hai-tao |
| contents | Large Language Models (LLMs) face computational inefficiencies and redundant processing when handling long context inputs, prompting a focus on compression techniques. While existing semantic vector-based compression methods achieve promising performance, these methods fail to account for the intrinsic information density variations between context chunks, instead allocating soft tokens uniformly across context chunks. This uniform distribution inevitably diminishes allocation to information-critical regions. To address this, we propose Dynamic Allocation of Soft Tokens (DAST), a simple yet effective method that leverages the LLM's intrinsic understanding of contextual relevance to guide compression. DAST combines perplexity-based local information with attention-driven global information to dynamically allocate soft tokens to the informative-rich chunks, enabling effective, context-aware compression. Experimental results across multiple benchmarks demonstrate that DAST surpasses state-of-the-art methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_11493 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DAST: Context-Aware Compression in LLMs via Dynamic Allocation of Soft Tokens Chen, Shaoshen Li, Yangning Xu, Zishan Li, Yinghui Su, Xin Shan, Zifei Zheng, Hai-tao Computation and Language Large Language Models (LLMs) face computational inefficiencies and redundant processing when handling long context inputs, prompting a focus on compression techniques. While existing semantic vector-based compression methods achieve promising performance, these methods fail to account for the intrinsic information density variations between context chunks, instead allocating soft tokens uniformly across context chunks. This uniform distribution inevitably diminishes allocation to information-critical regions. To address this, we propose Dynamic Allocation of Soft Tokens (DAST), a simple yet effective method that leverages the LLM's intrinsic understanding of contextual relevance to guide compression. DAST combines perplexity-based local information with attention-driven global information to dynamically allocate soft tokens to the informative-rich chunks, enabling effective, context-aware compression. Experimental results across multiple benchmarks demonstrate that DAST surpasses state-of-the-art methods. |
| title | DAST: Context-Aware Compression in LLMs via Dynamic Allocation of Soft Tokens |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.11493 |