Imagen de Portada

Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Fu, Zizhuo, Zeng, Wenxuan, Wang, Runsheng, Li, Meng
Formato:	Preprint
Publicado:	2026
Materias:	Computation and Language Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration
por: Yu, Zhongzhi, et al.
Publicado: (2024)

On the Existence and Behavior of Secondary Attention Sinks
por: Wong, Jeffrey T. H., et al.
Publicado: (2025)

How Attention Sinks Emerge in Large Language Models: An Interpretability Perspective
por: Peng, Runyu, et al.
Publicado: (2026)

Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms
por: Meng, Tian, et al.
Publicado: (2024)

Forgetting to Forget: Attention Sink as A Gateway for Backdooring LLM Unlearning
por: Shang, Bingqi, et al.
Publicado: (2025)

Attention Sinks as Internal Signals for Hallucination Detection in Large Language Models
por: Binkowski, Jakub, et al.
Publicado: (2026)

When Attention Sink Emerges in Language Models: An Empirical View
por: Gu, Xiangming, et al.
Publicado: (2024)

Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings
por: Zhang, Stephen, et al.
Publicado: (2025)

Attention Sinks in Massively Multilingual Neural Machine Translation:Discovery, Analysis, and Mitigation
por: Mutisya, Hillary, et al.
Publicado: (2026)

Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantization
por: Son, Seungwoo, et al.
Publicado: (2024)

Sink-Aware Pruning for Diffusion Language Models
por: Myrzakhan, Aidar, et al.
Publicado: (2026)

Does RoBERTa Perform Better than BERT in Continual Learning: An Attention Sink Perspective
por: Bai, Xueying, et al.
Publicado: (2024)

Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning
por: Sok, Jaewon, et al.
Publicado: (2026)

Attention Sinks and Outliers in Attention Residuals
por: Luo, Haozheng, et al.
Publicado: (2026)

PithTrain: A Compact and Agent-Native MoE Training System
por: Lai, Ruihang, et al.
Publicado: (2026)

CTR-Sink: Attention Sink for Language Models in Click-Through Rate Prediction
por: Li, Zixuan, et al.
Publicado: (2025)

The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks
por: Sun, Shangwen, et al.
Publicado: (2026)

LocMoE: A Low-Overhead MoE for Large Language Model Training
por: Li, Jing, et al.
Publicado: (2024)

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
por: Yuan, Jingyang, et al.
Publicado: (2025)

Improving Transformers with Dynamically Composable Multi-Head Attention
por: Xiao, Da, et al.
Publicado: (2024)

Mechanism and Emergence of Stacked Attention Heads in Multi-Layer Transformers
por: Musat, Tiberiu
Publicado: (2024)

Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning
por: Cao, Mingyu, et al.
Publicado: (2024)

MoE-Prism: Disentangling Monolithic Experts for Elastic MoE Services via Model-System Co-Designs
por: Xia, Xinfeng, et al.
Publicado: (2025)

Jakiro: Boosting Speculative Decoding with Decoupled Multi-Head via MoE
por: Huang, Haiduo, et al.
Publicado: (2025)

ASAP: Attention Sink Anchored Pruning
por: Lee, Jaehyuk, et al.
Publicado: (2026)

MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning
por: Manzoni, Andrea
Publicado: (2026)

Attention Sinks in Diffusion Language Models
por: Rulli, Maximo Eduardo, et al.
Publicado: (2025)

LayerBoost: Layer-Aware Attention Reduction for Efficient LLMs
por: Souibgui, Mohamed Ali, et al.
Publicado: (2026)

Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformers
por: Chen, Yihong, et al.
Publicado: (2026)

Faster MoE LLM Inference for Extremely Large Models
por: Yang, Haoqi, et al.
Publicado: (2025)

Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers
por: Ma, Wenhan, et al.
Publicado: (2025)

Efficient Streaming Language Models with Attention Sinks
por: Xiao, Guangxuan, et al.
Publicado: (2023)

Spectral Filters, Dark Signals, and Attention Sinks
por: Cancedda, Nicola
Publicado: (2024)

Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrast
por: Shi, Chufan, et al.
Publicado: (2024)

NOSA: Native and Offloadable Sparse Attention
por: Huang, Yuxiang, et al.
Publicado: (2025)

HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference
por: Zhong, Shuzhang, et al.
Publicado: (2025)

MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios
por: Li, Shuhuai, et al.
Publicado: (2026)

Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference
por: Liu, Baihui, et al.
Publicado: (2026)

Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference
por: Qiu, Quantong, et al.
Publicado: (2026)

GRIN: GRadient-INformed MoE
por: Liu, Liyuan, et al.
Publicado: (2024)