Copertina

The Structural Origin of Attention Sink: Variance Discrepancy, Super Neurons, and Dimension Disparity

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Li, Siquan, Jiang, Kaiqi, Sun, Jiacheng, Hu, Tianyang
Natura:	Preprint
Pubblicazione:	2026
Soggetti:	Machine Learning Artificial Intelligence
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Attention Sinks and Outliers in Attention Residuals
di: Luo, Haozheng, et al.
Pubblicazione: (2026)

Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformers
di: Chen, Yihong, et al.
Pubblicazione: (2026)

On the Existence and Behavior of Secondary Attention Sinks
di: Wong, Jeffrey T. H., et al.
Pubblicazione: (2025)

Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin
di: Queipo-de-Llano, Enrique, et al.
Pubblicazione: (2025)

On the Role of Preference Variance in Preference Optimization
di: Guo, Jiacheng, et al.
Pubblicazione: (2025)

When Attention Sink Emerges in Language Models: An Empirical View
di: Gu, Xiangming, et al.
Pubblicazione: (2024)

Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings
di: Zhang, Stephen, et al.
Pubblicazione: (2025)

Transformers Are Born Biased: Structural Inductive Biases at Random Initialization and Their Practical Consequences
di: Li, Siquan, et al.
Pubblicazione: (2026)

Neuronal Attention Circuit (NAC) for Representation Learning
di: Razzaq, Waleed, et al.
Pubblicazione: (2025)

A Minimum Variance Path Principle for Accurate and Stable Score-Based Density Ratio Estimation
di: Chen, Wei, et al.
Pubblicazione: (2026)

When Do Attention Circuits Form? Developmental Trajectories of Capability and Attention-Sink Emergence Across Three 1B-ClassArchitectures
di: Xu, Yongzhong
Pubblicazione: (2026)

SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Models
di: Liu, Junnan, et al.
Pubblicazione: (2026)

Streaming Attention Approximation via Discrepancy Theory
di: Kochetkova, Ekaterina, et al.
Pubblicazione: (2025)

Neuronal Stochastic Attention Circuit (NSAC) for Probabilistic Representation Learning
di: Razzaq, Waleed, et al.
Pubblicazione: (2026)

ProofAug: Efficient Neural Theorem Proving via Fine-grained Proof Structure Analysis
di: Liu, Haoxiong, et al.
Pubblicazione: (2025)

LATTLE: LLM Attention Transplant for Transfer Learning of Tabular Data Across Disparate Domains
di: Kowsar, Ibna, et al.
Pubblicazione: (2025)

Strassen Attention, Split VC Dimension and Compositionality in Transformers
di: Kozachinskiy, Alexander, et al.
Pubblicazione: (2025)

Graph Dimension Attention Networks for Enterprise Credit Assessment
di: Wei, Shaopeng, et al.
Pubblicazione: (2024)

Federated Unlearning in the Wild: Rethinking Fairness and Data Discrepancy
di: Huang, ZiHeng, et al.
Pubblicazione: (2025)

Memorization Sinks: Isolating Memorization during LLM Training
di: Ghosal, Gaurav R., et al.
Pubblicazione: (2025)

Ratio-Variance Regularized Policy Optimization for Efficient LLM Fine-tuning
di: Luo, Yu, et al.
Pubblicazione: (2026)

Decomposing Attention To Find Context-Sensitive Neurons
di: Gibson, Alex
Pubblicazione: (2025)

Dual Randomized Smoothing: Beyond Global Noise Variance
di: Sun, Chenhao, et al.
Pubblicazione: (2025)

GCFX: Generative Counterfactual Explanations for Deep Graph Models at the Model Level
di: Hu, Jinlong, et al.
Pubblicazione: (2026)

Detection vs. Execution: Single-Bucket Probes Miss Half the Mamba-2 State Sink
di: Jiang, Yuhang
Pubblicazione: (2026)

Self-Supervised Representation Learning for Geospatial Objects: A Survey
di: Chen, Yile, et al.
Pubblicazione: (2024)

A Variance-Reduced Cubic-Regularized Newton for Policy Optimization
di: Sun, Cheng, et al.
Pubblicazione: (2025)

Sink-Aware Pruning for Diffusion Language Models
di: Myrzakhan, Aidar, et al.
Pubblicazione: (2026)

On Vanishing Variance in Transformer Length Generalization
di: Li, Ruining, et al.
Pubblicazione: (2025)

Incorporating Domain Differential Equations into Graph Convolutional Networks to Lower Generalization Discrepancy
di: Sun, Yue, et al.
Pubblicazione: (2024)

Weighted Graph Structure Learning with Attention Denoising for Node Classification
di: Wang, Tingting, et al.
Pubblicazione: (2025)

Unveiling and Controlling Anomalous Attention Distribution in Transformers
di: Yan, Ruiqing, et al.
Pubblicazione: (2024)

FLUID: Continuous-Time Hyperconnected Sparse Transformer for Sink-Free Learning
di: Razzaq, Waleed, et al.
Pubblicazione: (2026)

Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention
di: Súkeník, Peter, et al.
Pubblicazione: (2026)

AdaFlow: Imitation Learning with Variance-Adaptive Flow-Based Policies
di: Hu, Xixi, et al.
Pubblicazione: (2024)

Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs
di: Huang, Luke J., et al.
Pubblicazione: (2026)

Bias Detection via Maximum Subgroup Discrepancy
di: Němeček, Jiří, et al.
Pubblicazione: (2025)

Discrepancy-Aware Graph Mask Auto-Encoder
di: Zheng, Ziyu, et al.
Pubblicazione: (2025)

Understanding Prediction Discrepancies in Machine Learning Classifiers
di: Renard, Xavier, et al.
Pubblicazione: (2021)

Online Drift Detection with Maximum Concept Discrepancy
di: Wan, Ke, et al.
Pubblicazione: (2024)