Cover Image

SWAA: Sliding Window Attention Adaptation for Efficient and Quality Preserving Long Context Processing

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Yu, Yijiong, Liu, Jiale, Wu, Qingyun, Wang, Huazheng, Pei, Ji
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Memory-Augmented Agent Training for Business Document Understanding
by: Liu, Jiale, et al.
Published: (2024)

Sliding Window Attention Training for Efficient Large Language Models
by: Fu, Zichuan, et al.
Published: (2025)

Do LLMs Really Think Step-by-step In Implicit Reasoning?
by: Yu, Yijiong
Published: (2024)

Density-aware Soft Context Compression with Semi-Dynamic Compression Ratio
by: Yu, Yijiong, et al.
Published: (2026)

$π$-Attention: Periodic Sparse Transformers for Efficient Long-Context Modeling
by: Liu, Dong, et al.
Published: (2025)

Live-Evo: Online Evolution of Agentic Memory from Continuous Feedback
by: Zhang, Yaolun, et al.
Published: (2026)

Training With "Paraphrasing the Original Text" Teaches LLM to Better Retrieve in Long-context Tasks
by: Yu, Yijiong, et al.
Published: (2023)

Efficient Context Scaling with LongCat ZigZag Attention
by: Zhang, Chen, et al.
Published: (2025)

Recurrent Context Compression: Efficiently Expanding the Context Window of LLM
by: Huang, Chensen, et al.
Published: (2024)

ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMs
by: Liu, Xin, et al.
Published: (2025)

Embodied LLM Agents Learn to Cooperate in Organized Teams
by: Guo, Xudong, et al.
Published: (2024)

Speculative Pipeline Decoding: Higher-Accruacy and Zero-Bubble Speculation via Pipeline Parallelism
by: Yu, Yijiong, et al.
Published: (2026)

LLM-RankFusion: Mitigating Intrinsic Inconsistency in LLM-based Ranking
by: Zeng, Yifan, et al.
Published: (2024)

CoCA: Fusing Position Embedding with Collinear Constrained Attention in Transformers for Long Context Window Extending
by: Zhu, Shiyi, et al.
Published: (2023)

Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modeling
by: Hu, Xiang, et al.
Published: (2024)

EvoPool: Evolutionary Programmatic Annotation for Label-Efficient Specialized Supervision
by: Xu, Tianyi, et al.
Published: (2026)

LongHeads: Multi-Head Attention is Secretly a Long Context Processor
by: Lu, Yi, et al.
Published: (2024)

OWL: Overcoming Window Length-Dependence in Speculative Decoding for Long-Context Inputs
by: Lee, Jaeseong, et al.
Published: (2025)

Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengths
by: Fu, Tianyu, et al.
Published: (2024)

ATACompressor: Adaptive Task-Aware Compression for Efficient Long-Context Processing in LLMs
by: Li, Xuancheng, et al.
Published: (2026)

Long-Context Generalization with Sparse Attention
by: Vasylenko, Pavlo, et al.
Published: (2025)

LongCodeBench: Evaluating Coding LLMs at 1M Context Windows
by: Rando, Stefano, et al.
Published: (2025)

SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modeling
by: Liu, Dong, et al.
Published: (2025)

LongSafety: Evaluating Long-Context Safety of Large Language Models
by: Lu, Yida, et al.
Published: (2025)

Offline Training of Language Model Agents with Functions as Learnable Weights
by: Zhang, Shaokun, et al.
Published: (2024)

LAWCAT: Efficient Distillation from Quadratic to Linear Attention with Convolution across Tokens for Long Context Modeling
by: Liu, Zeyu, et al.
Published: (2025)

Influence Guided Context Selection for Effective Retrieval-Augmented Generation
by: Deng, Jiale, et al.
Published: (2025)

MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling
by: MiniCPM Team, et al.
Published: (2026)

LiteLong: Resource-Efficient Long-Context Data Synthesis for LLMs
by: Jia, Junlong, et al.
Published: (2025)

Context Memorization for Efficient Long Context Generation
by: Okoshi, Yasuyuki, et al.
Published: (2026)

WAND: Windowed Attention and Knowledge Distillation for Efficient Autoregressive Text-to-Speech Models
by: Lee, Hanna, et al.
Published: (2026)

APCE: Adaptive Progressive Context Expansion for Long Context Processing
by: Lee, Baisub, et al.
Published: (2025)

PSC: Extending Context Window of Large Language Models via Phase Shift Calibration
by: Zhu, Wenqiao, et al.
Published: (2025)

Methodology of Adapting Large English Language Models for Specific Cultural Contexts
by: Zhang, Wenjing, et al.
Published: (2024)

Sentinel: Decoding Context Utilization via Attention Probing for Efficient LLM Context Compression
by: Zhang, Yong, et al.
Published: (2025)

MSWA: Refining Local Attention with Multi-ScaleWindow Attention
by: Xu, Yixing, et al.
Published: (2025)

Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning
by: Ling Team, et al.
Published: (2025)

Self-Taught Agentic Long Context Understanding
by: Zhuang, Yufan, et al.
Published: (2025)

Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts
by: Chen, Yingfa, et al.
Published: (2026)

SLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Window
by: Raunak, Vikas, et al.
Published: (2023)