Image de couverture de livre

A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Qiu, Zihan, Huang, Zeyu, Wen, Kaiyue, Jin, Peng, Zheng, Bo, Zhou, Yuxin, Huang, Haofeng, Wang, Zekun, Li, Xiao, Zhang, Huaqing, Xu, Yang, Lian, Haoran, Zhang, Siqi, Men, Rui, Zhang, Jianwei, Titov, Ivan, Liu, Dayiheng, Zhou, Jingren, Lin, Junyang
Format:	Preprint
Publié:	2026
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models
par: Qiu, Zihan, et autres
Publié: (2025)

Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
par: Qiu, Zihan, et autres
Publié: (2025)

Attention Sinks and Outliers in Attention Residuals
par: Luo, Haozheng, et autres
Publié: (2026)

Configuration-to-Performance Scaling Law with Neural Ansatz
par: Zhang, Huaqing, et autres
Publié: (2026)

SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
par: Tang, Shengkun, et autres
Publié: (2026)

Rethinking RoPE Scaling in Quantized LLM: Theory, Outlier, and Channel-Band Analysis with Weight Rescaling
par: Qiao, Ye, et autres
Publié: (2025)

Rotated Runtime Smooth: Training-Free Activation Smoother for accurate INT4 inference
par: Yi, Ke, et autres
Publié: (2024)

Invertible Residual Rescaling Models
par: Li, Jinmin, et autres
Publié: (2024)

ProcessBench: Identifying Process Errors in Mathematical Reasoning
par: Zheng, Chujie, et autres
Publié: (2024)

The Lessons of Developing Process Reward Models in Mathematical Reasoning
par: Zhang, Zhenru, et autres
Publié: (2025)

From Sparse Dependence to Sparse Attention: Unveiling How Chain-of-Thought Enhances Transformer Sample Efficiency
par: Wen, Kaiyue, et autres
Publié: (2024)

Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
par: Zhang, Yanzhao, et autres
Publié: (2025)

SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization
par: Zhang, Jintao, et autres
Publié: (2024)

Q-ROAR: Outlier-Aware Rescaling for RoPE Position Interpolation in Quantized Long-Context LLMs
par: Qiao, Ye, et autres
Publié: (2025)

Language Models can Self-Lengthen to Generate Long Texts
par: Quan, Shanghaoran, et autres
Publié: (2024)

P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs
par: Zhang, Yidan, et autres
Publié: (2024)

Layerwise Recurrent Router for Mixture-of-Experts
par: Qiu, Zihan, et autres
Publié: (2024)

Post-hoc Reward Calibration: A Case Study on Length Bias
par: Huang, Zeyu, et autres
Publié: (2024)

Canzona: A Unified, Asynchronous, and Load-Balanced Framework for Distributed Matrix-based Optimizers
par: Wang, Liangyu, et autres
Publié: (2026)

Extracting Visual Facts from Intermediate Layers for Mitigating Hallucinations in Multimodal Large Language Models
par: Zhou, Haoran, et autres
Publié: (2025)

HellaSwag-Pro: A Large-Scale Bilingual Benchmark for Evaluating the Robustness of LLMs in Commonsense Reasoning
par: Li, Xiaoyuan, et autres
Publié: (2025)

Task Generalization With AutoRegressive Compositional Structure: Can Learning From $D$ Tasks Generalize to $D^{T}$ Tasks?
par: Abedsoltan, Amirhesam, et autres
Publié: (2025)

Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking
par: Li, Mingxin, et autres
Publié: (2026)

Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free Real Image Editing
par: Titov, Vadim, et autres
Publié: (2024)

SCALE-LoRA: Auditing Post-Retrieval LoRA Composition with Residual Merging and View Reliability
par: Zhou, Shuaipeng, et autres
Publié: (2026)

Qwen2.5-Coder Technical Report
par: Hui, Binyuan, et autres
Publié: (2024)

Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach
par: Dong, Wei, et autres
Publié: (2024)

Bayesian Bandit Algorithms with Approximate Inference in Stochastic Linear Bandits
par: Huang, Ziyi, et autres
Publié: (2024)

Block-Based Multi-Scale Image Rescaling
par: Li, Jian, et autres
Publié: (2024)

Randomly Punctured Reed-Solomon Codes Achieve the List Decoding Capacity over Polynomial-Size Alphabets
par: Guo, Zeyu, et autres
Publié: (2023)

A Novel Self-Evolution Framework for Large Language Models
par: Sun, Haoran, et autres
Publié: (2025)

An Uncertainty-Driven Adaptive Self-Alignment Framework for Large Language Models
par: Sun, Haoran, et autres
Publié: (2025)

Preference-Aware Memory Update for Long-Term LLM Agents
par: Sun, Haoran, et autres
Publié: (2025)

When Attention Sink Emerges in Language Models: An Empirical View
par: Gu, Xiangming, et autres
Publié: (2024)

PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts
par: Wang, Yiming, et autres
Publié: (2025)

Quantum Metric-induced Oscillations in Flat Bands
par: Zeng, Hui, et autres
Publié: (2024)

Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling
par: Huang, Zeyu, et autres
Publié: (2025)

A Controllable Examination for Long-Context Language Models
par: Yang, Yijun, et autres
Publié: (2025)

SWE-Universe: Scale Real-World Verifiable Environments to Millions
par: Chen, Mouxiang, et autres
Publié: (2026)

Overcoming Joint Intractability with Lossless Hierarchical Speculative Decoding
par: Zhou, Yuxuan, et autres
Publié: (2026)