Imagen de Portada

Architecture-Dependent Processing Mode Dynamics in Transformer Attention: Opposing Transitions in MHA, GQA, and MoE Models

Fuente: Zenodo

Guardado en:

Detalles Bibliográficos
Autor principal:	Ichikawa, Yuki
Formato:	Recurso digital
Publicado:	Zenodo 2026
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Mixture of Attention Schemes (MoAS): Learning to Route Between MHA, GQA, and MQA
por: Gumaan, Esmail
Publicado: (2025)

Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA
por: Jin, Qingyun, et al.
Publicado: (2024)

Sensitivity-Positional Co-Localization in GQA Transformers
por: Rao, Manoj Chandrashekar
Publicado: (2026)

Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers
por: Chen, Anrui, et al.
Publicado: (2026)

GazeFormer-MoE: Context-Aware Gaze Estimation via CLIP and MoE Transformer
por: Zhao, Xinyuan, et al.
Publicado: (2026)

MiM-DiT: MoE in MoE with Diffusion Transformers for All-in-One Image Restoration
por: Kong, Lingshun, et al.
Publicado: (2026)

Dense2MoE: Restructuring Diffusion Transformer to MoE for Efficient Text-to-Image Generation
por: Zheng, Youwei, et al.
Publicado: (2025)

DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-training
por: Jin, Can, et al.
Publicado: (2025)

LLaDA-MoE: A Sparse MoE Diffusion Language Model
por: Zhu, Fengqi, et al.
Publicado: (2025)

MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancing
por: Ma, Haiyue, et al.
Publicado: (2025)

Muon: Training and Trade-offs with Latent Attention and MoE
por: Mehta, Sushant, et al.
Publicado: (2025)

Janus: Disaggregating Attention and Experts for Scalable MoE Inference
por: Zhang, Zhexiang, et al.
Publicado: (2025)

SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference
por: Chen, Liangkun, et al.
Publicado: (2025)

Optimizing MoE Routers: Design, Implementation, and Evaluation in Transformer Models
por: Harvey, Daniel Fidel, et al.
Publicado: (2025)

MoE-Prefill: Zero Redundancy Overheads in MoE Prefill Serving
por: Su, Zhaoyuan, et al.
Publicado: (2026)

MoE-PHDS: One MoE checkpoint for flexible runtime sparsity
por: Hannah, Lauren. A, et al.
Publicado: (2025)

MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?
por: Ma, Songkai, et al.
Publicado: (2025)

D$^{2}$MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving
por: Wang, Haodong, et al.
Publicado: (2025)

Revealing the Challenges of Attention-FFN Disaggregation for Modern MoE Models and Hardware Systems
por: Liu, Guowei, et al.
Publicado: (2026)

The MoE-Empowered Edge LLMs Deployment: Architecture, Challenges, and Opportunities
por: Li, Ning, et al.
Publicado: (2025)

STAMImputer: Spatio-Temporal Attention MoE for Traffic Data Imputation
por: Wang, Yiming, et al.
Publicado: (2025)

MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs
por: Cao, Shiyi, et al.
Publicado: (2024)

Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
por: Wu, Haoyuan, et al.
Publicado: (2025)

EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference
por: Qian, Yulei, et al.
Publicado: (2024)

GW-MoE: Resolving Uncertainty in MoE Router with Global Workspace Theory
por: Wu, Haoze, et al.
Publicado: (2024)

MoCHA: Advanced Vision-Language Reasoning with MoE Connector and Hierarchical Group Attention
por: Pang, Yuqi, et al.
Publicado: (2025)

CAT-MoEformer: Context-Aware Temporal MoE Transformer for Beam Prediction
por: Zhou, Changkai, et al.
Publicado: (2026)

KBVQ-MoE: KLT-guided SVD with Bias-Corrected Vector Quantization for MoE Large Language Models
por: Xu, Zukang, et al.
Publicado: (2026)

MoE-Prism: Disentangling Monolithic Experts for Elastic MoE Services via Model-System Co-Designs
por: Xia, Xinfeng, et al.
Publicado: (2025)

MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning
por: Manzoni, Andrea
Publicado: (2026)

LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashing
por: Nie, Xiaonan, et al.
Publicado: (2024)

OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
por: Wang, Liujianfu, et al.
Publicado: (2025)

Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs
por: Tang, Yehui, et al.
Publicado: (2025)

Spectral Manifold Regularization for Stable and Modular Routing in Deep MoE Architectures
por: Delibasoglu, Ibrahim
Publicado: (2026)

DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models
por: Aghdam, Maryam Akhavan, et al.
Publicado: (2024)

Harder Tasks Need More Experts: Dynamic Routing in MoE Models
por: Huang, Quzhe, et al.
Publicado: (2024)

Accelerating MoE with Dynamic In-Switch Computing on Multi-GPUs
por: Zhang, Qijun, et al.
Publicado: (2026)

MoE-GS: Mixture of Experts for Dynamic Gaussian Splatting
por: Jin, In-Hwan, et al.
Publicado: (2025)

Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs
por: Li, Bo, et al.
Publicado: (2026)

UniMoE-Audio: Unified Speech and Music Generation with Dynamic-Capacity MoE
por: Liu, Zhenyu, et al.
Publicado: (2025)