Copertina

The Effect of Attention Head Count on Transformer Approximation

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Yu, Penghao, Jiang, Haotian, Bao, Zeyu, Yu, Ruoxi, Li, Qianxiao
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

InfoFlow: A Framework for Multi-Layer Transformer Analysis
di: Yu, Penghao, et al.
Pubblicazione: (2026)

The Effect of Depth on the Expressivity of Deep Linear State-Space Models
di: Bao, Zeyu, et al.
Pubblicazione: (2025)

Allocation of Parameters in Transformers
di: Yu, Ruoxi, et al.
Pubblicazione: (2025)

Approximation Rate of the Transformer Architecture for Sequence Modeling
di: Jiang, Haotian, et al.
Pubblicazione: (2023)

Numerical Investigation of Sequence Modeling Theory using Controllable Memory Functions
di: Jiang, Haotian, et al.
Pubblicazione: (2025)

How Transformers Get Rich: Approximation and Dynamics Analysis
di: Wang, Mingze, et al.
Pubblicazione: (2024)

Inverse Approximation Theory for Nonlinear Recurrent Neural Networks
di: Wang, Shida, et al.
Pubblicazione: (2023)

Learning task-specific predictive models for scientific computing
di: Yin, Jianyuan, et al.
Pubblicazione: (2025)

From Generalization Analysis to Optimization Designs for State Space Models
di: Liu, Fusheng, et al.
Pubblicazione: (2024)

Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models
di: Liu, Fusheng, et al.
Pubblicazione: (2024)

Do Attention Heads Compete or Cooperate during Counting?
di: Zsámboki, Pál, et al.
Pubblicazione: (2025)

Accelerating Legacy Numerical Solvers by Non-intrusive Gradient-based Meta-solving
di: Arisaka, Sohei, et al.
Pubblicazione: (2024)

Unifying back-propagation and forward-forward algorithms through model predictive control
di: Ren, Lianhai, et al.
Pubblicazione: (2024)

Unveiling and Controlling Anomalous Attention Distribution in Transformers
di: Yan, Ruiqing, et al.
Pubblicazione: (2024)

Data Efficient Any Transformer-to-Mamba Distillation via Attention Bridge
di: Wang, Penghao, et al.
Pubblicazione: (2025)

Memorization Capacity of Multi-Head Attention in Transformers
di: Mahdavi, Sadegh, et al.
Pubblicazione: (2023)

Improving Transformers with Dynamically Composable Multi-Head Attention
di: Xiao, Da, et al.
Pubblicazione: (2024)

DynGMA: a robust approach for learning stochastic differential equations from data
di: Zhu, Aiqing, et al.
Pubblicazione: (2024)

Learning Macroscopic Dynamics from Partial Microscopic Observations
di: Chen, Mengyi, et al.
Pubblicazione: (2024)

DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion
di: Chen, Yilong, et al.
Pubblicazione: (2024)

StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterization
di: Wang, Shida, et al.
Pubblicazione: (2023)

Learning Permutation-invariant Macroscopic Dynamics
di: Han, Zhichao, et al.
Pubblicazione: (2026)

Approximate Gibbs Sampler for Efficient Inference of Hierarchical Bayesian Models for Grouped Count Data
di: Yu, Jin-Zhu, et al.
Pubblicazione: (2022)

Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers
di: Chen, Anrui, et al.
Pubblicazione: (2026)

On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis
di: Li, Zhong, et al.
Pubblicazione: (2020)

Counting in Small Transformers: The Delicate Interplay between Attention and Feed-Forward Layers
di: Behrens, Freya, et al.
Pubblicazione: (2024)

Interleaved Head Attention
di: Duvvuri, Sai Surya, et al.
Pubblicazione: (2026)

Machine Unlearning under Retain-Forget Entanglement
di: Cheng, Jingpu, et al.
Pubblicazione: (2026)

Prediction of Herd Life in Dairy Cows Using Multi-Head Attention Transformers
di: Saki, Mahdi, et al.
Pubblicazione: (2025)

Universal Approximation with Softmax Attention
di: Hu, Jerry Yao-Chieh, et al.
Pubblicazione: (2025)

ComplexFormer: Disruptively Advancing Transformer Inference Ability via Head-Specific Complex Vector Attention
di: Shao, Jintian, et al.
Pubblicazione: (2025)

Mechanism and Emergence of Stacked Attention Heads in Multi-Layer Transformers
di: Musat, Tiberiu
Pubblicazione: (2024)

Adaptive Head Budgeting for Efficient Multi-Head Attention
di: Faye, Bilal, et al.
Pubblicazione: (2026)

Efficient Attention via Pre-Scoring: Prioritizing Informative Keys in Transformers
di: Li, Zhexiang, et al.
Pubblicazione: (2025)

Beyond Parallelism: Synergistic Computational Graph Effects in Multi-Head Attention
di: Borde, Haitz Sáez de Ocáriz
Pubblicazione: (2025)

Continuity-Preserving Convolutional Autoencoders for Learning Continuous Latent Dynamical Models from Images
di: Zhu, Aiqing, et al.
Pubblicazione: (2025)

The Anxiety of Influence: Bloom Filters in Transformer Attention Heads
di: Balogh, Peter
Pubblicazione: (2026)

Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs
di: Guo, Tianyu, et al.
Pubblicazione: (2024)

ProxyAttn: Guided Sparse Attention via Representative Heads
di: Wang, Yixuan, et al.
Pubblicazione: (2025)

SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention
di: Csordás, Róbert, et al.
Pubblicazione: (2023)