Imagen de Portada

Adaptive Head Budgeting for Efficient Multi-Head Attention

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Faye, Bilal, Mbaye, Abdoulaye, Azzag, Hanane, Lebbah, Mustapha
Formato:	Preprint
Publicado:	2026
Materias:	Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Lightweight Modular Parameter-Efficient Tuning for Open-Vocabulary Object Detection
por: Faye, Bilal, et al.
Publicado: (2024)

Prototype-Guided Diffusion: Visual Conditioning without External Memory
por: Faye, Bilal, et al.
Publicado: (2025)

Supervised Batch Normalization
por: Faye, Bilal, et al.
Publicado: (2024)

Value-Free Policy Optimization via Reward Partitioning
por: Faye, Bilal, et al.
Publicado: (2025)

OneEncoder: A Lightweight Framework for Progressive Alignment of Modalities
por: Faye, Bilal, et al.
Publicado: (2024)

Unsupervised Adaptive Normalization
por: Faye, Bilal, et al.
Publicado: (2024)

Lightweight Cross-Modal Representation Learning
por: Faye, Bilal, et al.
Publicado: (2024)

Adaptative Context Normalization: A Boost for Deep Learning in Image Processing
por: Faye, Bilal, et al.
Publicado: (2024)

Enhancing Neural Network Representations with Prior Knowledge-Based Normalization
por: Faye, Bilal, et al.
Publicado: (2024)

Game Theory Meets Statistical Mechanics in Deep Learning Design
por: Bouchaffra, Djamel, et al.
Publicado: (2024)

Context Normalization Layer with Applications
por: Faye, Bilal, et al.
Publicado: (2023)

Energy-Regularized Spatial Masking: A Novel Approach to Enhancing Robustness and Interpretability in Vision Models
por: Devynck, Tom, et al.
Publicado: (2026)

Coalition Free Energy and Adaptive Precision in Multi-Agent Cooperation
por: Bouchaffra, Djamel, et al.
Publicado: (2026)

Distributed MCMC inference for Bayesian Non-Parametric Latent Block Model
por: Khoufache, Reda, et al.
Publicado: (2024)

MB-ORES: A Multi-Branch Object Reasoner for Visual Grounding in Remote Sensing
por: Radouane, Karim, et al.
Publicado: (2025)

NeuroGame Transformer: Gibbs-Inspired Attention Driven by Game Theory and Statistical Physics
por: Bouchaffra, Djamel, et al.
Publicado: (2026)

A Collective Variational Principle Unifying Bayesian Inference, Game Theory, and Thermodynamics
por: Bouchaffra, Djamel, et al.
Publicado: (2026)

Evaluating the Efficacy of Instance Incremental vs. Batch Learning in Delayed Label Environments: An Empirical Study on Tabular Data Streaming for Fraud Detection
por: Amekoe, Kodjo Mawuena, et al.
Publicado: (2024)

Manual Verbalizer Enrichment for Few-Shot Text Classification
por: Nguyen, Quang Anh, et al.
Publicado: (2024)

Multi-Head Low-Rank Attention
por: Liu, Songtao, et al.
Publicado: (2026)

MoH: Multi-Head Attention as Mixture-of-Head Attention
por: Jin, Peng, et al.
Publicado: (2024)

Memorization Capacity of Multi-Head Attention in Transformers
por: Mahdavi, Sadegh, et al.
Publicado: (2023)

Interleaved Head Attention
por: Duvvuri, Sai Surya, et al.
Publicado: (2026)

Efficient LLMs with AMP: Attention Heads and MLP Pruning
por: Mugnaini, Leandro Giusti, et al.
Publicado: (2025)

Dynamic Adaptive Shared Experts with Grouped Multi-Head Attention Mixture of Experts
por: Li, Cheng, et al.
Publicado: (2025)

DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion
por: Chen, Yilong, et al.
Publicado: (2024)

CHAI: Clustered Head Attention for Efficient LLM Inference
por: Agarwal, Saurabh, et al.
Publicado: (2024)

Unveiling Simplicities of Attention: Adaptive Long-Context Head Identification
por: Donhauser, Konstantin, et al.
Publicado: (2025)

Boosting House Price Estimations with Multi-Head Gated Attention
por: Sellam, Zakaria Abdellah, et al.
Publicado: (2024)

Geometric Analysis of Token Selection in Multi-Head Attention
por: Mudarisov, Timur, et al.
Publicado: (2026)

Improving Transformers with Dynamically Composable Multi-Head Attention
por: Xiao, Da, et al.
Publicado: (2024)

Superiority of Multi-Head Attention in In-Context Linear Regression
por: Cui, Yingqian, et al.
Publicado: (2024)

Benign Overfitting in Single-Head Attention
por: Magen, Roey, et al.
Publicado: (2024)

Multi-Head LatentMoE and Head Parallel: Communication-Efficient and Deterministic MoE Parallelism
por: Cui, Chenwei, et al.
Publicado: (2026)

Beyond Parallelism: Synergistic Computational Graph Effects in Multi-Head Attention
por: Borde, Haitz Sáez de Ocáriz
Publicado: (2025)

A Capacity-Based Rationale for Multi-Head Attention
por: Adler, Micah
Publicado: (2025)

Efficient Image Generation with Variadic Attention Heads
por: Walton, Steven, et al.
Publicado: (2022)

Multi-Head Spectral-Adaptive Graph Anomaly Detection
por: Cao, Qingyue, et al.
Publicado: (2025)

The Strong Lottery Ticket Hypothesis for Multi-Head Attention Mechanisms
por: Otsuka, Hikari, et al.
Publicado: (2025)

Mechanism and Emergence of Stacked Attention Heads in Multi-Layer Transformers
por: Musat, Tiberiu
Publicado: (2024)