Position IDs Matter: An Enhanced Position Layout for Efficient Context Compression in Large Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911177038102528 |
|---|---|
| author | Zhao, Runsong Liu, Xin Liu, Xinyu Huang, Pengcheng Xiao, Chunyang Xiao, Tong Zhu, Jingbo |
| author_facet | Zhao, Runsong Liu, Xin Liu, Xinyu Huang, Pengcheng Xiao, Chunyang Xiao, Tong Zhu, Jingbo |
| contents | Using special tokens (e.g., gist, memory, or compressed tokens) to compress context information is a common practice for large language models (LLMs). However, existing approaches often neglect that position encodings inherently induce local inductive biases in models, causing the compression process to ignore holistic contextual dependencies. We propose \textbf{Enhanced Position Layout (EPL)}, a simple yet effective method that improves the context compression capability of LLMs by only adjusting position IDs, the numerical identifiers that specify token positions. EPL minimizes the distance between context tokens and their corresponding special tokens and at the same time maintains the sequence order in position IDs between context tokens, special tokens, and the subsequent tokens. Integrating EPL into our best performing context compression model results in a 1.9 ROUGE-1 F1 improvement on out-of-domain question answering datasets on average. When extended to multimodal scenarios, EPL leads to an average accuracy gain of 2.6 points for vision compression LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_14364 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Position IDs Matter: An Enhanced Position Layout for Efficient Context Compression in Large Language Models Zhao, Runsong Liu, Xin Liu, Xinyu Huang, Pengcheng Xiao, Chunyang Xiao, Tong Zhu, Jingbo Computation and Language Using special tokens (e.g., gist, memory, or compressed tokens) to compress context information is a common practice for large language models (LLMs). However, existing approaches often neglect that position encodings inherently induce local inductive biases in models, causing the compression process to ignore holistic contextual dependencies. We propose \textbf{Enhanced Position Layout (EPL)}, a simple yet effective method that improves the context compression capability of LLMs by only adjusting position IDs, the numerical identifiers that specify token positions. EPL minimizes the distance between context tokens and their corresponding special tokens and at the same time maintains the sequence order in position IDs between context tokens, special tokens, and the subsequent tokens. Integrating EPL into our best performing context compression model results in a 1.9 ROUGE-1 F1 improvement on out-of-domain question answering datasets on average. When extended to multimodal scenarios, EPL leads to an average accuracy gain of 2.6 points for vision compression LLMs. |
| title | Position IDs Matter: An Enhanced Position Layout for Efficient Context Compression in Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2409.14364 |