Image de couverture de livre

Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modeling

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Hu, Xiang, Teng, Zhihao, Zhao, Jun, Wu, Wei, Tu, Kewei
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Hardware-aligned Hierarchical Sparse Attention for Efficient Long-term Memory Access
par: Hu, Xiang, et autres
Publié: (2025)

A Systematic Study of Compositional Syntactic Transformer Language Models
par: Zhao, Yida, et autres
Publié: (2025)

Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scale
par: Hu, Xiang, et autres
Publié: (2024)

DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Acceleration
par: Zhang, Hanzhi, et autres
Publié: (2025)

LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models
par: Zhao, Liang, et autres
Publié: (2024)

Augmenting Transformers with Recursively Composed Multi-grained Representations
par: Hu, Xiang, et autres
Publié: (2023)

Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Models
par: Zhao, Yida, et autres
Publié: (2024)

SEAL: Scaling to Emphasize Attention for Long-Context Retrieval
par: Lee, Changhun, et autres
Publié: (2025)

Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Models
par: Hu, Xiang, et autres
Publié: (2025)

Understanding and Improving Length Generalization in Hierarchical Sparse Attention Models
par: Leng, Jiaqi, et autres
Publié: (2025)

CAT: Causal Attention Tuning For Injecting Fine-grained Causal Knowledge into Large Language Models
par: Han, Kairong, et autres
Publié: (2025)

LongHeads: Multi-Head Attention is Secretly a Long Context Processor
par: Lu, Yi, et autres
Publié: (2024)

Efficient Context Scaling with LongCat ZigZag Attention
par: Zhang, Chen, et autres
Publié: (2025)

InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Models
par: Yan, Yuchen, et autres
Publié: (2025)

$π$-Attention: Periodic Sparse Transformers for Efficient Long-Context Modeling
par: Liu, Dong, et autres
Publié: (2025)

You Only Use Reactive Attention Slice For Long Context Retrieval
par: Soh, Yun Joon, et autres
Publié: (2024)

Correlation-Aware Select and Merge Attention for Efficient Fine-Tuning and Context Length Extension
par: Wang, Ning, et autres
Publié: (2024)

Retrieval meets Long Context Large Language Models
par: Xu, Peng, et autres
Publié: (2023)

CNNSum: Exploring Long-Context Summarization with Large Language Models in Chinese Novels
par: Wei, Lingxiao, et autres
Publié: (2024)

Learning Robust Named Entity Recognizers From Noisy Data With Retrieval Augmentation
par: Ai, Chaoyi, et autres
Publié: (2024)

SWAA: Sliding Window Attention Adaptation for Efficient and Quality Preserving Long Context Processing
par: Yu, Yijiong, et autres
Publié: (2025)

Large Language Models are In-Context Molecule Learners
par: Li, Jiatong, et autres
Publié: (2024)

Shifting Long-Context LLMs Research from Input to Output
par: Wu, Yuhao, et autres
Publié: (2025)

Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning
par: Ling Team, et autres
Publié: (2025)

TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection
par: Wu, Wei, et autres
Publié: (2024)

Flash Multi-Head Feed-Forward Network
par: Zhang, Minshen, et autres
Publié: (2025)

Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?
par: Lee, Jinhyuk, et autres
Publié: (2024)

From AR to Diffusion: Efficiently Adapting Large Language Models with Strictly Causal and Elastic Horizons
par: Ma, Xiangyu, et autres
Publié: (2026)

LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaboration
par: Zhao, Jun, et autres
Publié: (2024)

EcoSafeRAG: Efficient Security through Context Analysis in Retrieval-Augmented Generation
par: Yao, Ruobing, et autres
Publié: (2025)

Self-Demos: Eliciting Out-of-Demonstration Generalizability in Large Language Models
par: He, Wei, et autres
Publié: (2024)

Context Length Alone Hurts LLM Performance Despite Perfect Retrieval
par: Du, Yufeng, et autres
Publié: (2025)

When Long Helps Short: How Context Length in Supervised Fine-tuning Affects Behavior of Large Language Models
par: Zheng, Yingming, et autres
Publié: (2025)

Anti-Length Shift: Dynamic Outlier Truncation for Training Efficient Reasoning Models
par: Wu, Wei, et autres
Publié: (2026)

Long-Context Generalization with Sparse Attention
par: Vasylenko, Pavlo, et autres
Publié: (2025)

MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling
par: MiniCPM Team, et autres
Publié: (2026)

LAWCAT: Efficient Distillation from Quadratic to Linear Attention with Convolution across Tokens for Long Context Modeling
par: Liu, Zeyu, et autres
Publié: (2025)

Context-Aware Initialization for Reducing Generative Path Length in Diffusion Language Models
par: Miao, Tongyuan, et autres
Publié: (2025)

AttentionRAG: Attention-Guided Context Pruning in Retrieval-Augmented Generation
par: Fang, Yixiong, et autres
Publié: (2025)

SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modeling
par: Liu, Dong, et autres
Publié: (2025)