Imagen de Portada

DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Chen, Yilong, Zhang, Linhao, Shang, Junyuan, Zhang, Zhenyu, Liu, Tingwen, Wang, Shuohuan, Sun, Yu
Formato:	Preprint
Publicado:	2024
Materias:	Machine Learning Artificial Intelligence Computation and Language
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformation
por: Chen, Yilong, et al.
Publicado: (2026)

MoR: Mixture of Ranks for Low-Rank Adaptation Tuning
por: Tang, Chuanyu, et al.
Publicado: (2024)

Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking
por: Chen, Yilong, et al.
Publicado: (2025)

Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Looping
por: Chen, Yao, et al.
Publicado: (2026)

Safety Alignment Should Be Made More Than Just A Few Attention Heads
por: Huang, Chao, et al.
Publicado: (2025)

Mixture of Hidden-Dimensions Transformer
por: Chen, Yilong, et al.
Publicado: (2024)

Debiasing LLMs by Masking Unfairness-Driving Attention Heads
por: Han, Tingxu, et al.
Publicado: (2025)

NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time
por: Chen, Yilong, et al.
Publicado: (2024)

LongHeads: Multi-Head Attention is Secretly a Long Context Processor
por: Lu, Yi, et al.
Publicado: (2024)

Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging
por: Hui, Tingfeng, et al.
Publicado: (2024)

Decoupling Knowledge and Reasoning in Transformers: A Modular Architecture with Generalized Cross-Attention
por: Guo, Zhenyu, et al.
Publicado: (2025)

The Anxiety of Influence: Bloom Filters in Transformer Attention Heads
por: Balogh, Peter
Publicado: (2026)

Sycophancy Hides Linearly in the Attention Heads
por: Genadi, Rifo, et al.
Publicado: (2026)

Jakiro: Boosting Speculative Decoding with Decoupled Multi-Head via MoE
por: Huang, Haiduo, et al.
Publicado: (2025)

Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs
por: Ji, Tao, et al.
Publicado: (2025)

Which Attention Heads Matter for In-Context Learning?
por: Yin, Kayo, et al.
Publicado: (2025)

RecurFormer: Not All Transformer Heads Need Self-Attention
por: Yan, Ruiqing, et al.
Publicado: (2024)

K-ON: Stacking Knowledge On the Head Layer of Large Language Model
por: Guo, Lingbing, et al.
Publicado: (2025)

Less Is More: Fast and Accurate Reasoning with Cross-Head Unified Sparse Attention
por: Yang, Lijie, et al.
Publicado: (2025)

DESTEIN: Navigating Detoxification of Language Models via Universal Steering Pairs and Head-wise Activation Fusion
por: Li, Yu, et al.
Publicado: (2024)

On the Role of Attention Heads in Large Language Model Safety
por: Zhou, Zhenhong, et al.
Publicado: (2024)

Identifying Semantic Induction Heads to Understand In-Context Learning
por: Ren, Jie, et al.
Publicado: (2024)

Focusing on Language: Revealing and Exploiting Language Attention Heads in Multilingual Large Language Models
por: Liu, Xin, et al.
Publicado: (2025)

Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts
por: Gu, Naibin, et al.
Publicado: (2025)

Analyzing Multi-Head Attention on Trojan BERT Models
por: Wang, Jingwei
Publicado: (2024)

Latent Multi-Head Attention for Small Language Models
por: Mehta, Sushant, et al.
Publicado: (2025)

Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning
por: Zhang, Yichi, et al.
Publicado: (2024)

Learning to Route Queries to Heads for Attention-based Re-ranking with Large Language Models
por: Tian, Yuxing, et al.
Publicado: (2026)

NoVo: Norm Voting off Hallucinations with Attention Heads in Large Language Models
por: Ho, Zheng Yi, et al.
Publicado: (2024)

A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputs
por: Shelmanov, Artem, et al.
Publicado: (2025)

Adaptive Data Augmentation for Aspect Sentiment Quad Prediction
por: Zhang, Wenyuan, et al.
Publicado: (2024)

SEAL: Steerable Reasoning Calibration of Large Language Models for Free
por: Chen, Runjin, et al.
Publicado: (2025)

JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention
por: Tian, Yuandong, et al.
Publicado: (2023)

On the Emergence of Induction Heads for In-Context Learning
por: Musat, Tiberiu, et al.
Publicado: (2025)

KID: Knowledge-Injected Dual-Head Learning for Knowledge-Grounded Harmful Meme Detection
por: Li, Yaocong, et al.
Publicado: (2026)

LycheeDecode: Accelerating Long-Context LLM Inference via Hybrid-Head Sparse Decoding
por: Lin, Gang, et al.
Publicado: (2026)

Assortment of Attention Heads: Accelerating Federated PEFT with Head Pruning and Strategic Client Selection
por: Venkatesha, Yeshwanth, et al.
Publicado: (2025)

Ranking Unraveled: Recipes for LLM Rankings in Head-to-Head AI Combat
por: Daynauth, Roland, et al.
Publicado: (2024)

Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning
por: Fu, Yu, et al.
Publicado: (2024)

Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers
por: Chen, Siyu, et al.
Publicado: (2024)