Imagen de Portada

Improving Transformers with Dynamically Composable Multi-Head Attention

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Xiao, Da, Meng, Qingye, Li, Shengping, Yuan, Xingyuan
Formato:	Preprint
Publicado:	2024
Materias:	Machine Learning Computation and Language
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections
por: Xiao, Da, et al.
Publicado: (2025)

Benchmarking and Understanding Compositional Relational Reasoning of LLMs
por: Ni, Ruikang, et al.
Publicado: (2024)

Mechanism and Emergence of Stacked Attention Heads in Multi-Layer Transformers
por: Musat, Tiberiu
Publicado: (2024)

Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
por: Fu, Zizhuo, et al.
Publicado: (2026)

SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention
por: Csordás, Róbert, et al.
Publicado: (2023)

Dynamic Rank Reinforcement Learning for Adaptive Low-Rank Multi-Head Self Attention in Large Language Models
por: Erden, Caner
Publicado: (2025)

The Anxiety of Influence: Bloom Filters in Transformer Attention Heads
por: Balogh, Peter
Publicado: (2026)

CNCast: Leveraging 3D Swin Transformer and DiT for Enhanced Regional Weather Forecasting
por: Liang, Hongli, et al.
Publicado: (2025)

Head Pursuit: Probing Attention Specialization in Multimodal Transformers
por: Basile, Lorenzo, et al.
Publicado: (2025)

TensorLLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMs
por: Gu, Yuxuan, et al.
Publicado: (2025)

RecurFormer: Not All Transformer Heads Need Self-Attention
por: Yan, Ruiqing, et al.
Publicado: (2024)

DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion
por: Chen, Yilong, et al.
Publicado: (2024)

Selective Attention Improves Transformer
por: Leviathan, Yaniv, et al.
Publicado: (2024)

Supervised Fine-Tuning Achieve Rapid Task Adaption Via Alternating Attention Head Activation Patterns
por: Zhao, Yang, et al.
Publicado: (2024)

ComplexFormer: Disruptively Advancing Transformer Inference Ability via Head-Specific Complex Vector Attention
por: Shao, Jintian, et al.
Publicado: (2025)

CHAI: Clustered Head Attention for Efficient LLM Inference
por: Agarwal, Saurabh, et al.
Publicado: (2024)

FlowCast-ODE: Continuous Hourly Weather Forecasting with Dynamic Flow Matching and ODE Solver
por: He, Shuangshuang, et al.
Publicado: (2025)

GLU Attention Improve Transformer
por: Wang, Zehao
Publicado: (2025)

Multi-Head Attention Is a Multi-Player Game
por: Chakrabarti, Kushal, et al.
Publicado: (2026)

Limits of Transformer Language Models on Learning to Compose Algorithms
por: Thomm, Jonathan, et al.
Publicado: (2024)

Sparse Query Attention (SQA): A Computationally Efficient Attention Mechanism with Query Heads Reduction
por: Filipek, Adam
Publicado: (2025)

ProxyAttn: Guided Sparse Attention via Representative Heads
por: Wang, Yixuan, et al.
Publicado: (2025)

Unveiling Simplicities of Attention: Adaptive Long-Context Head Identification
por: Donhauser, Konstantin, et al.
Publicado: (2025)

ChemAmp: Amplified Chemistry Tools via Composable Agents
por: Li, Zhucong, et al.
Publicado: (2025)

RazorAttention: Efficient KV Cache Compression Through Retrieval Heads
por: Tang, Hanlin, et al.
Publicado: (2024)

Composable Interventions for Language Models
por: Kolbeinsson, Arinbjorn, et al.
Publicado: (2024)

Learnable Multi-Scale Wavelet Transformer: A Novel Alternative to Self-Attention
por: Kiruluta, Andrew, et al.
Publicado: (2025)

LASER: Attention with Exponential Transformation
por: Duvvuri, Sai Surya, et al.
Publicado: (2024)

Listening to the Wise Few: Select-and-Copy Attention Heads for Multiple-Choice QA
por: Tulchinskii, Eduard, et al.
Publicado: (2024)

Beyond Self Attention: A Subquadratic Fourier Wavelet Transformer with Multi Modal Fusion
por: Kiruluta, Andrew, et al.
Publicado: (2021)

ALIEN: Aligned Entropy Head for Improving Uncertainty Estimation of LLMs
por: Zabolotnyi, Artem, et al.
Publicado: (2025)

From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformers
por: Ildiz, M. Emrullah, et al.
Publicado: (2024)

Which Attention Heads Matter for In-Context Learning?
por: Yin, Kayo, et al.
Publicado: (2025)

Multi-matrix Factorization Attention
por: Hu, Jingcheng, et al.
Publicado: (2024)

Dynamic Attention-Guided Context Decoding for Mitigating Context Faithfulness Hallucinations in Large Language Models
por: Huang, Yanwen, et al.
Publicado: (2025)

SpectR: Dynamically Composing LM Experts with Spectral Routing
por: Fleshman, William, et al.
Publicado: (2025)

Generalized Probabilistic Attention Mechanism in Transformers
por: Heo, DongNyeong, et al.
Publicado: (2024)

Can Language Models Compose Skills In-Context?
por: Liu, Zidong, et al.
Publicado: (2025)

Selective Attention: Enhancing Transformer through Principled Context Control
por: Zhang, Xuechen, et al.
Publicado: (2024)

The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
por: Nawrot, Piotr, et al.
Publicado: (2025)