SepSeq: A Training-Free Framework for Long Numerical Sequence Processing in LLMs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915926468722688 |
|---|---|
| author | Sun, Jie Liu, Yu Han, Lu Deng, Qiwen Shu, Xiang Xiao, Yang Lu, Xingyu Zhou, Jun Liu, Pengfei Ma, Lintao Wu, Jiancan Wang, Xiang |
| author_facet | Sun, Jie Liu, Yu Han, Lu Deng, Qiwen Shu, Xiang Xiao, Yang Lu, Xingyu Zhou, Jun Liu, Pengfei Ma, Lintao Wu, Jiancan Wang, Xiang |
| contents | While transformer-based Large Language Models (LLMs) theoretically support massive context windows, they suffer from severe performance degradation when processing long numerical sequences. We attribute this failure to the attention dispersion in the Softmax mechanism, which prevents the model from concentrating attention. To overcome this, we propose Separate Sequence (SepSeq), a training-free, plug-and-play framework to mitigate dispersion by strategically inserting separator tokens. Mechanistically, we demonstrate that separator tokens act as an attention sink, recalibrating attention to focus on local segments while preserving global context. Extensive evaluations on 9 widely-adopted LLMs confirm the effectiveness of our approach: SepSeq yields an average relative accuracy improvement of 35.6% across diverse domains while reducing total inference token consumption by 16.4% on average. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_07737 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SepSeq: A Training-Free Framework for Long Numerical Sequence Processing in LLMs Sun, Jie Liu, Yu Han, Lu Deng, Qiwen Shu, Xiang Xiao, Yang Lu, Xingyu Zhou, Jun Liu, Pengfei Ma, Lintao Wu, Jiancan Wang, Xiang Computation and Language While transformer-based Large Language Models (LLMs) theoretically support massive context windows, they suffer from severe performance degradation when processing long numerical sequences. We attribute this failure to the attention dispersion in the Softmax mechanism, which prevents the model from concentrating attention. To overcome this, we propose Separate Sequence (SepSeq), a training-free, plug-and-play framework to mitigate dispersion by strategically inserting separator tokens. Mechanistically, we demonstrate that separator tokens act as an attention sink, recalibrating attention to focus on local segments while preserving global context. Extensive evaluations on 9 widely-adopted LLMs confirm the effectiveness of our approach: SepSeq yields an average relative accuracy improvement of 35.6% across diverse domains while reducing total inference token consumption by 16.4% on average. |
| title | SepSeq: A Training-Free Framework for Long Numerical Sequence Processing in LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.07737 |