Imagen de Portada

Explicit Multi-head Attention for Inter-head Interaction in Large Language Models

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Peng, Runyu, Zhou, Yunhua, Song, Demin, Lv, Kai, Wang, Bo, Guo, Qipeng, Qiu, Xipeng
Formato:	Preprint
Publicado:	2026
Materias:	Machine Learning Artificial Intelligence Computation and Language
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

How Attention Sinks Emerge in Large Language Models: An Interpretability Perspective
por: Peng, Runyu, et al.
Publicado: (2026)

Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning
por: Zhang, Yechen, et al.
Publicado: (2026)

Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections
por: Wang, Bo, et al.
Publicado: (2025)

Data-free Weight Compress and Denoise for Large Language Models
por: Peng, Runyu, et al.
Publicado: (2024)

ReAttention: Training-Free Infinite Context with Finite Attention Scope
por: Liu, Xiaoran, et al.
Publicado: (2024)

How to Set the Learning Rate for Large-Scale Pre-training?
por: Zhou, Yunhua, et al.
Publicado: (2026)

Evolution of Concepts in Language Model Pre-Training
por: Ge, Xuyang, et al.
Publicado: (2025)

BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments
por: Wang, Xinghao, et al.
Publicado: (2024)

How to Set the Batch Size for Large-Scale Pre-training?
por: Zhou, Yunhua, et al.
Publicado: (2026)

GTA: Grouped-head latenT Attention
por: Sun, Luoyang, et al.
Publicado: (2025)

Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs
por: Ji, Tao, et al.
Publicado: (2025)

Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance
por: Ye, Jiasheng, et al.
Publicado: (2024)

MARRO: Multi-headed Attention for Rhetorical Role Labeling in Legal Documents
por: Bambroo, Purbid, et al.
Publicado: (2025)

Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?
por: Zeng, Zhiyuan, et al.
Publicado: (2025)

Turn Waste into Worth: Rectifying Top-$k$ Router of MoE
por: Zeng, Zhiyuan, et al.
Publicado: (2024)

DuoDecoding: Hardware-aware Heterogeneous Speculative Decoding with Dynamic Multi-Sequence Drafting
por: Lv, Kai, et al.
Publicado: (2025)

Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data
por: Guo, Xu, et al.
Publicado: (2026)

Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Models
por: Cao, Jiaqi, et al.
Publicado: (2025)

Emergent Structured Representations Support Flexible In-Context Inference in Large Language Models
por: Xu, Ningyu, et al.
Publicado: (2026)

Amphista: Bi-directional Multi-head Decoding for Accelerating LLM Inference
por: Li, Zeping, et al.
Publicado: (2024)

GAProtoNet: A Multi-head Graph Attention-based Prototypical Network for Interpretable Text Classification
por: Wen, Ximing, et al.
Publicado: (2024)

Identifying Semantic Induction Heads to Understand In-Context Learning
por: Ren, Jie, et al.
Publicado: (2024)

Hymba: A Hybrid-head Architecture for Small Language Models
por: Dong, Xin, et al.
Publicado: (2024)

Evaluating the Performance of Large Language Models on GAOKAO Benchmark
por: Zhang, Xiaotian, et al.
Publicado: (2023)

From Implicit to Explicit: Enhancing Self-Recognition in Large Language Models
por: Zhou, Yinghan, et al.
Publicado: (2025)

Has Your Pretrained Model Improved? A Multi-head Posterior Based Approach
por: Aboagye, Prince, et al.
Publicado: (2024)

Full Parameter Fine-tuning for Large Language Models with Limited Resources
por: Lv, Kai, et al.
Publicado: (2023)

World-aware Planning Narratives Enhance Large Vision-Language Model Planner
por: Shi, Junhao, et al.
Publicado: (2025)

Context-level Language Modeling by Learning Predictive Context Embeddings
por: Dai, Beiya, et al.
Publicado: (2025)

Next Concept Prediction in Discrete Latent Space Leads to Stronger Language Models
por: Liu, Yuliang, et al.
Publicado: (2026)

Multi-head Temporal Latent Attention
por: Deng, Keqi, et al.
Publicado: (2025)

Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debate
por: Xiong, Kai, et al.
Publicado: (2023)

Code-driven Number Sequence Calculation: Enhancing the inductive Reasoning Abilities of Large Language Models
por: Chen, Kedi, et al.
Publicado: (2025)

Prism: Spectral-Aware Block-Sparse Attention
por: Wang, Xinghao, et al.
Publicado: (2026)

Synergetic Event Understanding: A Collaborative Approach to Cross-Document Event Coreference Resolution with Large Language Models
por: Min, Qingkai, et al.
Publicado: (2024)

Focusing on Language: Revealing and Exploiting Language Attention Heads in Multilingual Large Language Models
por: Liu, Xin, et al.
Publicado: (2025)

Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspective
por: Zeng, Zhiyuan, et al.
Publicado: (2024)

Out-of-distribution generalization via composition: a lens through induction heads in Transformers
por: Song, Jiajun, et al.
Publicado: (2024)

Balanced Data Sampling for Language Model Training with Clustering
por: Shao, Yunfan, et al.
Publicado: (2024)

Safe Inputs but Unsafe Output: Benchmarking Cross-modality Safety Alignment of Large Vision-Language Model
por: Wang, Siyin, et al.
Publicado: (2024)