Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2511.08128 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912701809164288 |
|---|---|
| author | Tarasov, Dmitrii Goncharova, Elizaveta Andrey, Kuznetsov |
| author_facet | Tarasov, Dmitrii Goncharova, Elizaveta Andrey, Kuznetsov |
| contents | This work investigates context compression for Large Language Models (LLMs) using learned compression tokens to reduce the memory and computational demands of processing long sequences. We demonstrate that pre-trained LLMs can be fine-tuned to compress their context by factors of 2x to 8x without significant performance degradation, as evaluated on both short-context and long-context benchmarks. Furthermore, in experiments on a 3-billion-parameter LLaMA model, our method achieves results on par with alternative compression techniques while attaining higher compression ratios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_08128 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Sentence-Anchored Gist Compression for Long-Context LLMs Tarasov, Dmitrii Goncharova, Elizaveta Andrey, Kuznetsov Computation and Language This work investigates context compression for Large Language Models (LLMs) using learned compression tokens to reduce the memory and computational demands of processing long sequences. We demonstrate that pre-trained LLMs can be fine-tuned to compress their context by factors of 2x to 8x without significant performance degradation, as evaluated on both short-context and long-context benchmarks. Furthermore, in experiments on a 3-billion-parameter LLaMA model, our method achieves results on par with alternative compression techniques while attaining higher compression ratios. |
| title | Sentence-Anchored Gist Compression for Long-Context LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2511.08128 |