GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignment
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915782523355136 |
|---|---|
| author | Tang, Jiwei Zhang, Zhicheng Wu, Shunlong Ye, Jingheng Bai, Lichen Wang, Zitai Lu, Tingwei Hai, Lin Zhao, Yiming Zheng, Hai-Tao Kim, Hong-Gee |
| author_facet | Tang, Jiwei Zhang, Zhicheng Wu, Shunlong Ye, Jingheng Bai, Lichen Wang, Zitai Lu, Tingwei Hai, Lin Zhao, Yiming Zheng, Hai-Tao Kim, Hong-Gee |
| contents | Large Language Models (LLMs) have achieved remarkable performance across a wide range of Natural Language Processing (NLP) tasks. However, in long-context scenarios, they face two challenges: high computational cost and information redundancy. To address these challenges, we propose GMSA, an encoder-decoder context compression framework that generates a compact sequence of soft tokens for downstream tasks. GMSA introduces Group Merging to achieve more uniform aggregation, mitigating semantic dominance during autoencoder pretraining, and Layer Semantic Alignment (LSA) to bridge the semantic gap between high-level abstract semantics and low-level input semantics. We first pretrain GMSA as an autoencoder and then fine-tune it for downstream tasks. Experiments demonstrate that GMSA improves context reconstruction compared to existing soft prompt compression paradigm and outperforms baselines on multiple long-context question answering and summarization benchmarks across two backbone models, while maintaining low end-to-end latency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_12215 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignment Tang, Jiwei Zhang, Zhicheng Wu, Shunlong Ye, Jingheng Bai, Lichen Wang, Zitai Lu, Tingwei Hai, Lin Zhao, Yiming Zheng, Hai-Tao Kim, Hong-Gee Computation and Language Large Language Models (LLMs) have achieved remarkable performance across a wide range of Natural Language Processing (NLP) tasks. However, in long-context scenarios, they face two challenges: high computational cost and information redundancy. To address these challenges, we propose GMSA, an encoder-decoder context compression framework that generates a compact sequence of soft tokens for downstream tasks. GMSA introduces Group Merging to achieve more uniform aggregation, mitigating semantic dominance during autoencoder pretraining, and Layer Semantic Alignment (LSA) to bridge the semantic gap between high-level abstract semantics and low-level input semantics. We first pretrain GMSA as an autoencoder and then fine-tune it for downstream tasks. Experiments demonstrate that GMSA improves context reconstruction compared to existing soft prompt compression paradigm and outperforms baselines on multiple long-context question answering and summarization benchmarks across two backbone models, while maintaining low end-to-end latency. |
| title | GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignment |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.12215 |