Cover Image

Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformers

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Chen, Yihong, Lin, Zhouchen, Yao, Quanming
Format:	Preprint
Published:	2026
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Attention Sinks and Outliers in Attention Residuals
by: Luo, Haozheng, et al.
Published: (2026)

The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks
by: Sun, Shangwen, et al.
Published: (2026)

On the Existence and Behavior of Secondary Attention Sinks
by: Wong, Jeffrey T. H., et al.
Published: (2025)

Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention
by: Qiu, Haiquan, et al.
Published: (2025)

SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Models
by: Liu, Junnan, et al.
Published: (2026)

When Attention Sink Emerges in Language Models: An Empirical View
by: Gu, Xiangming, et al.
Published: (2024)

Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin
by: Queipo-de-Llano, Enrique, et al.
Published: (2025)

Softpick: No Attention Sink, No Massive Activations with Rectified Softmax
by: Zuhri, Zayd M. K., et al.
Published: (2025)

The Structural Origin of Attention Sink: Variance Discrepancy, Super Neurons, and Dimension Disparity
by: Li, Siquan, et al.
Published: (2026)

FLUID: Continuous-Time Hyperconnected Sparse Transformer for Sink-Free Learning
by: Razzaq, Waleed, et al.
Published: (2026)

Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings
by: Zhang, Stephen, et al.
Published: (2025)

Memorization Sinks: Isolating Memorization during LLM Training
by: Ghosal, Gaurav R., et al.
Published: (2025)

Sink-Aware Pruning for Diffusion Language Models
by: Myrzakhan, Aidar, et al.
Published: (2026)

When Do Attention Circuits Form? Developmental Trajectories of Capability and Attention-Sink Emergence Across Three 1B-ClassArchitectures
by: Xu, Yongzhong
Published: (2026)

Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention
by: Súkeník, Peter, et al.
Published: (2026)

GradMAP: Gradient-Based Multi-Agent Proximal Learning for Grid-Edge Flexibility
by: Zhou, Yihong, et al.
Published: (2026)

Adjusting the Output of Decision Transformer with Action Gradient
by: Lin, Rui, et al.
Published: (2025)

Zero-Shot RTL Code Generation with Attention Sink Augmented Large Language Models
by: Sandal, Selim, et al.
Published: (2024)

Quantum Gradient Class Activation Map for Model Interpretability
by: Lin, Hsin-Yi, et al.
Published: (2024)

Stochastic Parroting in Temporal Attention -- Regulating the Diagonal Sink
by: Hankemeier, Victoria, et al.
Published: (2026)

Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformers
by: Liang, Yingyu, et al.
Published: (2024)

Attention Saturation and Gradient Suppression at Inflection Layers: Diagnosing and Mitigating Bottlenecks in Transformer Adaptation
by: Zixian, Wang
Published: (2025)

HeatGen: A Guided Diffusion Framework for Multiphysics Heat Sink Design Optimization
by: Keramati, Hadi, et al.
Published: (2025)

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
by: Shin, Seungjun, et al.
Published: (2025)

On the Limitations and Capabilities of Position Embeddings for Length Generalization
by: Chen, Yang, et al.
Published: (2025)

Efficient Search for Customized Activation Functions with Gradient Descent
by: Strack, Lukas, et al.
Published: (2024)

Accurate and interpretable drug-drug interaction prediction enabled by knowledge subgraph learning
by: Wang, Yaqing, et al.
Published: (2023)

PACIA: Parameter-Efficient Adapter for Few-Shot Molecular Property Prediction
by: Wu, Shiguang, et al.
Published: (2023)

Graph Unitary Message Passing
by: Qiu, Haiquan, et al.
Published: (2024)

Detection vs. Execution: Single-Bucket Probes Miss Half the Mamba-2 State Sink
by: Jiang, Yuhang
Published: (2026)

Gradient Boosting within a Single Attention Layer
by: Sargolzaei, Saleh
Published: (2026)

GradientStabilizer:Fix the Norm, Not the Gradient
by: Huang, Tianjin, et al.
Published: (2025)

DIGIC: Domain Generalizable Imitation Learning by Causal Discovery
by: Chen, Yang, et al.
Published: (2024)

GFPack++: Improving 2D Irregular Packing by Learning Gradient Field with Attention
by: Xue, Tianyang, et al.
Published: (2024)

Gradient Regularized Natural Gradients
by: Dash, Satya Prakash, et al.
Published: (2026)

Learning Discrete Autoregressive Priors with Wasserstein Gradient Flow
by: Zheng, Bowen, et al.
Published: (2026)

The Reciprocity Gradient
by: Lin, Yue, et al.
Published: (2026)

G-Drift MIA: Membership Inference via Gradient-Induced Feature Drift in LLMs
by: Ranjan, Ravi, et al.
Published: (2026)

Massive Activations in Graph Neural Networks: Decoding Attention for Domain-Dependent Interpretability
by: Bini, Lorenzo, et al.
Published: (2024)

Relational Learning in Pre-Trained Models: A Theory from Hypergraph Recovery Perspective
by: Chen, Yang, et al.
Published: (2024)