Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2506.08371 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916790079062016 |
|---|---|
| author | Xiao, Zikai Wang, Ziyang Ma, Wen Zhang, Yan Shen, Wei Wang, Yan Gong, Luqi Liu, Zuozhu |
| author_facet | Xiao, Zikai Wang, Ziyang Ma, Wen Zhang, Yan Shen, Wei Wang, Yan Gong, Luqi Liu, Zuozhu |
| contents | While Large Language Models (LLMs) support long contexts, they struggle with performance degradation within the context window. Current solutions incur prohibitive training costs, leaving statistical behaviors and cost-effective approaches underexplored. From the decoding perspective, we identify the Posterior Salience Attenuation (PSA) phenomenon, where the salience ratio correlates with long-text performance degradation. Notably, despite the attenuation, gold tokens still occupy high-ranking positions in the decoding space. Motivated by it, we propose the training-free Positional Contrastive Decoding (PCD) that contrasts the logits derived from long-aware attention with those from designed local-aware attention, enabling the model to focus on the gains introduced by large-scale short-to-long training. Through the analysis of long-term decay simulation, we demonstrate that PCD effectively alleviates attention score degradation. Experimental results show that PCD achieves state-of-the-art performance on long-context benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_08371 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding Xiao, Zikai Wang, Ziyang Ma, Wen Zhang, Yan Shen, Wei Wang, Yan Gong, Luqi Liu, Zuozhu Computation and Language While Large Language Models (LLMs) support long contexts, they struggle with performance degradation within the context window. Current solutions incur prohibitive training costs, leaving statistical behaviors and cost-effective approaches underexplored. From the decoding perspective, we identify the Posterior Salience Attenuation (PSA) phenomenon, where the salience ratio correlates with long-text performance degradation. Notably, despite the attenuation, gold tokens still occupy high-ranking positions in the decoding space. Motivated by it, we propose the training-free Positional Contrastive Decoding (PCD) that contrasts the logits derived from long-aware attention with those from designed local-aware attention, enabling the model to focus on the gains introduced by large-scale short-to-long training. Through the analysis of long-term decay simulation, we demonstrate that PCD effectively alleviates attention score degradation. Experimental results show that PCD achieves state-of-the-art performance on long-context benchmarks. |
| title | Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.08371 |