Image de couverture de livre

Mixture of Attention Schemes (MoAS): Learning to Route Between MHA, GQA, and MQA

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteur principal:	Gumaan, Esmail
Format:	Preprint
Publié:	2025
Sujets:	Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

ExpertRAG: Efficient RAG with Mixture of Experts -- Optimizing Context Retrieval for Adaptive LLM Responses
par: Gumaan, Esmail
Publié: (2025)

Universal Approximation Theorem for a Single-Layer Transformer
par: Gumaan, Esmail
Publié: (2025)

Theoretical Foundations and Mitigation of Hallucination in Large Language Models
par: Gumaan, Esmail
Publié: (2025)

Galvatron: Automatic Distributed Training for Large Transformer Models
par: Gumaan, Esmail
Publié: (2025)

Architecture-Dependent Processing Mode Dynamics in Transformer Attention: Opposing Transitions in MHA, GQA, and MoE Models
par: Ichikawa, Yuki
Publié: (2026)

Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA
par: Jin, Qingyun, et autres
Publié: (2024)

Sensitivity-Positional Co-Localization in GQA Transformers
par: Rao, Manoj Chandrashekar
Publié: (2026)

A-MHA*: Anytime Multi-Heuristic A*
par: Natarajan, Ramkumar, et autres
Publié: (2025)

RouteMoA: Dynamic Routing without Pre-Inference Boosts Efficient Mixture-of-Agents
par: Wang, Jize, et autres
Publié: (2026)

GQA-μP: The maximal parameterization update for grouped query attention
par: Chickering, Kyle R., et autres
Publié: (2026)

ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Experts
par: Zhao, Heng, et autres
Publié: (2026)

MHA-RAG: Improving Efficiency, Accuracy, and Consistency by Encoding Exemplars as Soft Prompts
par: Jain, Abhinav, et autres
Publié: (2025)

AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Models
par: Zeng, Zihao, et autres
Publié: (2024)

LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Experts
par: Zhuang, Yuan, et autres
Publié: (2025)

SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmark
par: Liang, Zhenwen, et autres
Publié: (2024)

MoH: Multi-Head Attention as Mixture-of-Head Attention
par: Jin, Peng, et autres
Publié: (2024)

Time-MQA: Time Series Multi-Task Question Answering with Context Enhancement
par: Kong, Yaxuan, et autres
Publié: (2025)

MoBA: Mixture of Block Attention for Long-Context LLMs
par: Lu, Enzhe, et autres
Publié: (2025)

PediatricsMQA: a Multi-modal Pediatrics Question Answering Benchmark
par: Bahaj, Adil, et autres
Publié: (2025)

Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts
par: Hua, Yongxiang, et autres
Publié: (2025)

Attention-MoA: Enhancing Mixture-of-Agents via Inter-Agent Semantic Attention and Deep Residual Synthesis
par: Wen, Jianyu, et autres
Publié: (2026)

Whisper-MLA: Reducing GPU Memory Consumption of ASR Models based on MHA2MLA Conversion
par: Zhang, Sen, et autres
Publié: (2026)

R2GQA: Retriever-Reader-Generator Question Answering System to Support Students Understanding Legal Regulations in Higher Education
par: Do, Phuc-Tinh Pham, et autres
Publié: (2024)

DynMoLE: Boosting Mixture of LoRA Experts Fine-Tuning with a Hybrid Routing Mechanism
par: Li, Dengchun, et autres
Publié: (2025)

BadMoE: Backdooring Mixture-of-Experts LLMs via Optimizing Routing Triggers and Infecting Dormant Experts
par: Wang, Qingyue, et autres
Publié: (2025)

RouteHijack: Routing-Aware Attack on Mixture-of-Experts LLMs
par: Xu, Zhiyuan, et autres
Publié: (2026)

DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
par: Feng, Jiarui, et autres
Publié: (2026)

MoA: Mixture-of-Attention for Subject-Context Disentanglement in Personalized Image Generation
par: Wang, Kuan-Chieh, et autres
Publié: (2024)

Mixture-of-Depths Attention
par: Zhu, Lianghui, et autres
Publié: (2026)

Multilingual Routing in Mixture-of-Experts
par: Bandarkar, Lucas, et autres
Publié: (2025)

Routing-Free Mixture-of-Experts
par: Liu, Yilun, et autres
Publié: (2026)

Geometric Routing Enables Causal Expert Control in Mixture of Experts
par: Ternovtsii, Ivan, et autres
Publié: (2026)

MoEMeta: Mixture-of-Experts Meta Learning for Few-Shot Relational Learning
par: Wu, Han, et autres
Publié: (2025)

MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learning
par: Shen, Jingyan, et autres
Publié: (2025)

Geometric Mixture-of-Experts with Curvature-Guided Adaptive Routing for Graph Representation Learning
par: Cao, Haifang, et autres
Publié: (2026)

MHA2MLA-VLM: Enabling DeepSeek's Economical Multi-Head Latent Attention across Vision-Language Models
par: Fan, Xiaoran, et autres
Publié: (2026)

The Routing and Filtering Structure of Attention
par: Jamil, Shafayeth, et autres
Publié: (2026)

ECG-MoE: Mixture-of-Expert Electrocardiogram Foundation Model
par: Xu, Yuhao, et autres
Publié: (2026)

Mixture of Layers with Hybrid Attention
par: Ternovtsii, Ivan, et autres
Publié: (2026)

MoE-Loco: Mixture of Experts for Multitask Locomotion
par: Huang, Runhan, et autres
Publié: (2025)